2026-03-09 18:32:38,082 p=31156 u=zuul n=ansible | Starting galaxy collection install process 2026-03-09 18:32:38,083 p=31156 u=zuul n=ansible | Process install dependency map 2026-03-09 18:32:54,673 p=31156 u=zuul n=ansible | Starting collection install process 2026-03-09 18:32:54,674 p=31156 u=zuul n=ansible | Installing 'cifmw.general:1.0.0+2d3d99e4' to '/home/zuul/.ansible/collections/ansible_collections/cifmw/general' 2026-03-09 18:32:55,110 p=31156 u=zuul n=ansible | Created collection for cifmw.general:1.0.0+2d3d99e4 at /home/zuul/.ansible/collections/ansible_collections/cifmw/general 2026-03-09 18:32:55,110 p=31156 u=zuul n=ansible | cifmw.general:1.0.0+2d3d99e4 was installed successfully 2026-03-09 18:32:55,110 p=31156 u=zuul n=ansible | Installing 'containers.podman:1.16.2' to '/home/zuul/.ansible/collections/ansible_collections/containers/podman' 2026-03-09 18:32:55,169 p=31156 u=zuul n=ansible | Created collection for containers.podman:1.16.2 at /home/zuul/.ansible/collections/ansible_collections/containers/podman 2026-03-09 18:32:55,169 p=31156 u=zuul n=ansible | containers.podman:1.16.2 was installed successfully 2026-03-09 18:32:55,169 p=31156 u=zuul n=ansible | Installing 'community.general:10.0.1' to '/home/zuul/.ansible/collections/ansible_collections/community/general' 2026-03-09 18:32:55,846 p=31156 u=zuul n=ansible | Created collection for community.general:10.0.1 at /home/zuul/.ansible/collections/ansible_collections/community/general 2026-03-09 18:32:55,846 p=31156 u=zuul n=ansible | community.general:10.0.1 was installed successfully 2026-03-09 18:32:55,846 p=31156 u=zuul n=ansible | Installing 'ansible.posix:1.6.2' to '/home/zuul/.ansible/collections/ansible_collections/ansible/posix' 2026-03-09 18:32:55,886 p=31156 u=zuul n=ansible | Created collection for ansible.posix:1.6.2 at /home/zuul/.ansible/collections/ansible_collections/ansible/posix 2026-03-09 18:32:55,886 p=31156 u=zuul n=ansible | ansible.posix:1.6.2 was installed successfully 2026-03-09 18:32:55,886 p=31156 u=zuul n=ansible | Installing 'ansible.utils:5.1.2' to '/home/zuul/.ansible/collections/ansible_collections/ansible/utils' 2026-03-09 18:32:55,957 p=31156 u=zuul n=ansible | Created collection for ansible.utils:5.1.2 at /home/zuul/.ansible/collections/ansible_collections/ansible/utils 2026-03-09 18:32:55,957 p=31156 u=zuul n=ansible | ansible.utils:5.1.2 was installed successfully 2026-03-09 18:32:55,957 p=31156 u=zuul n=ansible | Installing 'community.libvirt:1.3.0' to '/home/zuul/.ansible/collections/ansible_collections/community/libvirt' 2026-03-09 18:32:55,974 p=31156 u=zuul n=ansible | Created collection for community.libvirt:1.3.0 at /home/zuul/.ansible/collections/ansible_collections/community/libvirt 2026-03-09 18:32:55,974 p=31156 u=zuul n=ansible | community.libvirt:1.3.0 was installed successfully 2026-03-09 18:32:55,974 p=31156 u=zuul n=ansible | Installing 'community.crypto:2.22.3' to '/home/zuul/.ansible/collections/ansible_collections/community/crypto' 2026-03-09 18:32:56,070 p=31156 u=zuul n=ansible | Created collection for community.crypto:2.22.3 at /home/zuul/.ansible/collections/ansible_collections/community/crypto 2026-03-09 18:32:56,070 p=31156 u=zuul n=ansible | community.crypto:2.22.3 was installed successfully 2026-03-09 18:32:56,070 p=31156 u=zuul n=ansible | Installing 'kubernetes.core:5.0.0' to '/home/zuul/.ansible/collections/ansible_collections/kubernetes/core' 2026-03-09 18:32:56,155 p=31156 u=zuul n=ansible | Created collection for kubernetes.core:5.0.0 at /home/zuul/.ansible/collections/ansible_collections/kubernetes/core 2026-03-09 18:32:56,155 p=31156 u=zuul n=ansible | kubernetes.core:5.0.0 was installed successfully 2026-03-09 18:32:56,155 p=31156 u=zuul n=ansible | Installing 'ansible.netcommon:7.1.0' to '/home/zuul/.ansible/collections/ansible_collections/ansible/netcommon' 2026-03-09 18:32:56,363 p=31156 u=zuul n=ansible | Created collection for ansible.netcommon:7.1.0 at /home/zuul/.ansible/collections/ansible_collections/ansible/netcommon 2026-03-09 18:32:56,363 p=31156 u=zuul n=ansible | ansible.netcommon:7.1.0 was installed successfully 2026-03-09 18:32:56,363 p=31156 u=zuul n=ansible | Installing 'openstack.config_template:2.1.1' to '/home/zuul/.ansible/collections/ansible_collections/openstack/config_template' 2026-03-09 18:32:56,379 p=31156 u=zuul n=ansible | Created collection for openstack.config_template:2.1.1 at /home/zuul/.ansible/collections/ansible_collections/openstack/config_template 2026-03-09 18:32:56,379 p=31156 u=zuul n=ansible | openstack.config_template:2.1.1 was installed successfully 2026-03-09 18:32:56,379 p=31156 u=zuul n=ansible | Installing 'junipernetworks.junos:9.1.0' to '/home/zuul/.ansible/collections/ansible_collections/junipernetworks/junos' 2026-03-09 18:32:56,557 p=31156 u=zuul n=ansible | Created collection for junipernetworks.junos:9.1.0 at /home/zuul/.ansible/collections/ansible_collections/junipernetworks/junos 2026-03-09 18:32:56,557 p=31156 u=zuul n=ansible | junipernetworks.junos:9.1.0 was installed successfully 2026-03-09 18:32:56,557 p=31156 u=zuul n=ansible | Installing 'cisco.ios:9.0.3' to '/home/zuul/.ansible/collections/ansible_collections/cisco/ios' 2026-03-09 18:32:56,738 p=31156 u=zuul n=ansible | Created collection for cisco.ios:9.0.3 at /home/zuul/.ansible/collections/ansible_collections/cisco/ios 2026-03-09 18:32:56,738 p=31156 u=zuul n=ansible | cisco.ios:9.0.3 was installed successfully 2026-03-09 18:32:56,738 p=31156 u=zuul n=ansible | Installing 'mellanox.onyx:1.0.0' to '/home/zuul/.ansible/collections/ansible_collections/mellanox/onyx' 2026-03-09 18:32:56,768 p=31156 u=zuul n=ansible | Created collection for mellanox.onyx:1.0.0 at /home/zuul/.ansible/collections/ansible_collections/mellanox/onyx 2026-03-09 18:32:56,768 p=31156 u=zuul n=ansible | mellanox.onyx:1.0.0 was installed successfully 2026-03-09 18:32:56,768 p=31156 u=zuul n=ansible | Installing 'community.okd:4.0.0' to '/home/zuul/.ansible/collections/ansible_collections/community/okd' 2026-03-09 18:32:56,792 p=31156 u=zuul n=ansible | Created collection for community.okd:4.0.0 at /home/zuul/.ansible/collections/ansible_collections/community/okd 2026-03-09 18:32:56,793 p=31156 u=zuul n=ansible | community.okd:4.0.0 was installed successfully 2026-03-09 18:32:56,793 p=31156 u=zuul n=ansible | Installing '@NAMESPACE@.@NAME@:3.1.4' to '/home/zuul/.ansible/collections/ansible_collections/@NAMESPACE@/@NAME@' 2026-03-09 18:32:56,854 p=31156 u=zuul n=ansible | Created collection for @NAMESPACE@.@NAME@:3.1.4 at /home/zuul/.ansible/collections/ansible_collections/@NAMESPACE@/@NAME@ 2026-03-09 18:32:56,854 p=31156 u=zuul n=ansible | @NAMESPACE@.@NAME@:3.1.4 was installed successfully 2026-03-09 18:33:04,080 p=31718 u=zuul n=ansible | PLAY [Bootstrap playbook] ****************************************************** 2026-03-09 18:33:04,098 p=31718 u=zuul n=ansible | TASK [Gathering Facts ] ******************************************************** 2026-03-09 18:33:04,099 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:04 +0000 (0:00:00.034) 0:00:00.034 ********** 2026-03-09 18:33:04,099 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:04 +0000 (0:00:00.033) 0:00:00.033 ********** 2026-03-09 18:33:05,219 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:33:05,237 p=31718 u=zuul n=ansible | TASK [cifmw_setup : Set custom cifmw PATH reusable fact cifmw_path={{ ansible_user_dir }}/.crc/bin:{{ ansible_user_dir }}/.crc/bin/oc:{{ ansible_user_dir }}/bin:{{ ansible_env.PATH }}, cacheable=True] *** 2026-03-09 18:33:05,238 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:01.139) 0:00:01.173 ********** 2026-03-09 18:33:05,238 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:01.139) 0:00:01.172 ********** 2026-03-09 18:33:05,262 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:33:05,270 p=31718 u=zuul n=ansible | TASK [cifmw_setup : Get customized parameters ci_framework_params={{ hostvars[inventory_hostname] | dict2items | selectattr("key", "match", "^(cifmw|pre|post)_(?!install_yamls|openshift_token|openshift_login|openshift_kubeconfig).*") | list | items2dict }}] *** 2026-03-09 18:33:05,270 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.032) 0:00:01.206 ********** 2026-03-09 18:33:05,270 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.032) 0:00:01.204 ********** 2026-03-09 18:33:05,343 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:33:05,349 p=31718 u=zuul n=ansible | TASK [install_ca : Ensure target directory exists path={{ cifmw_install_ca_trust_dir }}, state=directory, mode=0755] *** 2026-03-09 18:33:05,349 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.079) 0:00:01.285 ********** 2026-03-09 18:33:05,349 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.079) 0:00:01.283 ********** 2026-03-09 18:33:05,713 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:33:05,723 p=31718 u=zuul n=ansible | TASK [install_ca : Install internal CA from url url={{ cifmw_install_ca_url }}, dest={{ cifmw_install_ca_trust_dir }}, validate_certs={{ cifmw_install_ca_url_validate_certs | default(omit) }}, mode=0644] *** 2026-03-09 18:33:05,723 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.373) 0:00:01.658 ********** 2026-03-09 18:33:05,723 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.373) 0:00:01.657 ********** 2026-03-09 18:33:05,739 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:05,747 p=31718 u=zuul n=ansible | TASK [install_ca : Install custom CA bundle from inline dest={{ cifmw_install_ca_trust_dir }}/cifmw_inline_ca_bundle.crt, content={{ cifmw_install_ca_bundle_inline }}, mode=0644] *** 2026-03-09 18:33:05,749 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.026) 0:00:01.684 ********** 2026-03-09 18:33:05,749 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.026) 0:00:01.683 ********** 2026-03-09 18:33:05,767 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:05,773 p=31718 u=zuul n=ansible | TASK [install_ca : Install custom CA bundle from file dest={{ cifmw_install_ca_trust_dir }}/{{ cifmw_install_ca_bundle_src | basename }}, src={{ cifmw_install_ca_bundle_src }}, mode=0644] *** 2026-03-09 18:33:05,773 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.024) 0:00:01.709 ********** 2026-03-09 18:33:05,773 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.024) 0:00:01.708 ********** 2026-03-09 18:33:05,792 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:05,799 p=31718 u=zuul n=ansible | TASK [install_ca : Update ca bundle _raw_params=update-ca-trust] *************** 2026-03-09 18:33:05,799 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.026) 0:00:01.735 ********** 2026-03-09 18:33:05,799 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.026) 0:00:01.734 ********** 2026-03-09 18:33:05,820 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:05,834 p=31718 u=zuul n=ansible | TASK [repo_setup : Ensure directories are present path={{ cifmw_repo_setup_basedir }}/{{ item }}, state=directory, mode=0755] *** 2026-03-09 18:33:05,834 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.035) 0:00:01.770 ********** 2026-03-09 18:33:05,834 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:05 +0000 (0:00:00.035) 0:00:01.769 ********** 2026-03-09 18:33:06,091 p=31718 u=zuul n=ansible | changed: [controller] => (item=tmp) 2026-03-09 18:33:06,716 p=31718 u=zuul n=ansible | changed: [controller] => (item=artifacts/repositories) 2026-03-09 18:33:06,924 p=31718 u=zuul n=ansible | changed: [controller] => (item=venv/repo_setup) 2026-03-09 18:33:06,932 p=31718 u=zuul n=ansible | TASK [repo_setup : Make sure git-core package is installed name=git-core, state=present] *** 2026-03-09 18:33:06,932 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:06 +0000 (0:00:01.097) 0:00:02.868 ********** 2026-03-09 18:33:06,932 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:06 +0000 (0:00:01.097) 0:00:02.867 ********** 2026-03-09 18:33:07,963 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:33:07,973 p=31718 u=zuul n=ansible | TASK [repo_setup : Get repo-setup repository accept_hostkey=True, dest={{ cifmw_repo_setup_basedir }}/tmp/repo-setup, repo={{ cifmw_repo_setup_src }}] *** 2026-03-09 18:33:07,973 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:07 +0000 (0:00:01.040) 0:00:03.909 ********** 2026-03-09 18:33:07,973 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:07 +0000 (0:00:01.040) 0:00:03.907 ********** 2026-03-09 18:33:08,838 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:08,845 p=31718 u=zuul n=ansible | TASK [repo_setup : Initialize python venv and install requirements virtualenv={{ cifmw_repo_setup_venv }}, requirements={{ cifmw_repo_setup_basedir }}/tmp/repo-setup/requirements.txt, virtualenv_command=python3 -m venv --system-site-packages --upgrade-deps] *** 2026-03-09 18:33:08,845 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:08 +0000 (0:00:00.871) 0:00:04.780 ********** 2026-03-09 18:33:08,845 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:08 +0000 (0:00:00.871) 0:00:04.779 ********** 2026-03-09 18:33:16,773 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:16,779 p=31718 u=zuul n=ansible | TASK [repo_setup : Install repo-setup package chdir={{ cifmw_repo_setup_basedir }}/tmp/repo-setup, creates={{ cifmw_repo_setup_venv }}/bin/repo-setup, _raw_params={{ cifmw_repo_setup_venv }}/bin/python setup.py install] *** 2026-03-09 18:33:16,779 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:16 +0000 (0:00:07.934) 0:00:12.715 ********** 2026-03-09 18:33:16,779 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:16 +0000 (0:00:07.934) 0:00:12.714 ********** 2026-03-09 18:33:17,630 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:17,638 p=31718 u=zuul n=ansible | TASK [repo_setup : Set cifmw_repo_setup_dlrn_hash_tag from content provider cifmw_repo_setup_dlrn_hash_tag={{ content_provider_dlrn_md5_hash }}] *** 2026-03-09 18:33:17,638 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:17 +0000 (0:00:00.858) 0:00:13.574 ********** 2026-03-09 18:33:17,638 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:17 +0000 (0:00:00.859) 0:00:13.573 ********** 2026-03-09 18:33:17,663 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:17,669 p=31718 u=zuul n=ansible | TASK [repo_setup : Run repo-setup _raw_params={{ cifmw_repo_setup_venv }}/bin/repo-setup {{ cifmw_repo_setup_promotion }} {{ cifmw_repo_setup_additional_repos }} -d {{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }} -b {{ cifmw_repo_setup_branch }} --rdo-mirror {{ cifmw_repo_setup_rdo_mirror }} {% if cifmw_repo_setup_dlrn_hash_tag | length > 0 %} --dlrn-hash-tag {{ cifmw_repo_setup_dlrn_hash_tag }} {% endif %} -o {{ cifmw_repo_setup_output }}] *** 2026-03-09 18:33:17,669 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:17 +0000 (0:00:00.031) 0:00:13.605 ********** 2026-03-09 18:33:17,669 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:17 +0000 (0:00:00.031) 0:00:13.604 ********** 2026-03-09 18:33:18,295 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:18,303 p=31718 u=zuul n=ansible | TASK [repo_setup : Get component repo url={{ cifmw_repo_setup_dlrn_uri }}/{{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }}-{{ cifmw_repo_setup_branch }}/component/{{ cifmw_repo_setup_component_name }}/{{ cifmw_repo_setup_component_promotion_tag }}/delorean.repo, dest={{ cifmw_repo_setup_output }}/{{ cifmw_repo_setup_component_name }}_{{ cifmw_repo_setup_component_promotion_tag }}_delorean.repo, mode=0644] *** 2026-03-09 18:33:18,304 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:18 +0000 (0:00:00.634) 0:00:14.239 ********** 2026-03-09 18:33:18,304 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:18 +0000 (0:00:00.634) 0:00:14.238 ********** 2026-03-09 18:33:18,338 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:18,346 p=31718 u=zuul n=ansible | TASK [repo_setup : Rename component repo path={{ cifmw_repo_setup_output }}/{{ cifmw_repo_setup_component_name }}_{{ cifmw_repo_setup_component_promotion_tag }}_delorean.repo, regexp=delorean-component-{{ cifmw_repo_setup_component_name }}, replace={{ cifmw_repo_setup_component_name }}-{{ cifmw_repo_setup_component_promotion_tag }}] *** 2026-03-09 18:33:18,347 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:18 +0000 (0:00:00.043) 0:00:14.282 ********** 2026-03-09 18:33:18,347 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:18 +0000 (0:00:00.043) 0:00:14.281 ********** 2026-03-09 18:33:18,374 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:18,382 p=31718 u=zuul n=ansible | TASK [repo_setup : Disable component repo in current-podified dlrn repo path={{ cifmw_repo_setup_output }}/delorean.repo, section=delorean-component-{{ cifmw_repo_setup_component_name }}, option=enabled, value=0, mode=0644] *** 2026-03-09 18:33:18,382 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:18 +0000 (0:00:00.035) 0:00:14.318 ********** 2026-03-09 18:33:18,382 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:18 +0000 (0:00:00.035) 0:00:14.316 ********** 2026-03-09 18:33:18,407 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:18,416 p=31718 u=zuul n=ansible | TASK [repo_setup : Run repo-setup-get-hash _raw_params={{ cifmw_repo_setup_venv }}/bin/repo-setup-get-hash --dlrn-url {{ cifmw_repo_setup_dlrn_uri[:-1] }} --os-version {{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }} --release {{ cifmw_repo_setup_branch }} {% if cifmw_repo_setup_component_name | length > 0 -%} --component {{ cifmw_repo_setup_component_name }} --tag {{ cifmw_repo_setup_component_promotion_tag }} {% else -%} --tag {{cifmw_repo_setup_promotion }} {% endif -%} {% if (cifmw_repo_setup_dlrn_hash_tag | length > 0) and (cifmw_repo_setup_component_name | length <= 0) -%} --dlrn-hash-tag {{ cifmw_repo_setup_dlrn_hash_tag }} {% endif -%} --json] *** 2026-03-09 18:33:18,416 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:18 +0000 (0:00:00.033) 0:00:14.352 ********** 2026-03-09 18:33:18,416 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:18 +0000 (0:00:00.033) 0:00:14.350 ********** 2026-03-09 18:33:18,857 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:18,863 p=31718 u=zuul n=ansible | TASK [repo_setup : Dump full hash in delorean.repo.md5 file content={{ _repo_setup_json['full_hash'] }} , dest={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5, mode=0644] *** 2026-03-09 18:33:18,863 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:18 +0000 (0:00:00.447) 0:00:14.799 ********** 2026-03-09 18:33:18,863 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:18 +0000 (0:00:00.446) 0:00:14.797 ********** 2026-03-09 18:33:19,670 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:19,677 p=31718 u=zuul n=ansible | TASK [repo_setup : Dump current-podified hash url={{ cifmw_repo_setup_dlrn_uri }}/{{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }}-{{ cifmw_repo_setup_branch }}/current-podified/delorean.repo.md5, dest={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5, mode=0644] *** 2026-03-09 18:33:19,677 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.814) 0:00:15.613 ********** 2026-03-09 18:33:19,677 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.814) 0:00:15.612 ********** 2026-03-09 18:33:19,688 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:19,695 p=31718 u=zuul n=ansible | TASK [repo_setup : Slurp current podified hash src={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5] *** 2026-03-09 18:33:19,695 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.631 ********** 2026-03-09 18:33:19,695 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.629 ********** 2026-03-09 18:33:19,706 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:19,712 p=31718 u=zuul n=ansible | TASK [repo_setup : Update the value of full_hash _repo_setup_json={{ _repo_setup_json | combine({'full_hash': _hash}, recursive=true) }}] *** 2026-03-09 18:33:19,712 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.648 ********** 2026-03-09 18:33:19,712 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.646 ********** 2026-03-09 18:33:19,723 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:19,729 p=31718 u=zuul n=ansible | TASK [repo_setup : Export hashes facts for further use cifmw_repo_setup_full_hash={{ _repo_setup_json['full_hash'] }}, cifmw_repo_setup_commit_hash={{ _repo_setup_json['commit_hash'] }}, cifmw_repo_setup_distro_hash={{ _repo_setup_json['distro_hash'] }}, cifmw_repo_setup_extended_hash={{ _repo_setup_json['extended_hash'] }}, cifmw_repo_setup_dlrn_api_url={{ _repo_setup_json['dlrn_api_url'] }}, cifmw_repo_setup_dlrn_url={{ _repo_setup_json['dlrn_url'] }}, cifmw_repo_setup_release={{ _repo_setup_json['release'] }}, cacheable=True] *** 2026-03-09 18:33:19,729 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.665 ********** 2026-03-09 18:33:19,729 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.664 ********** 2026-03-09 18:33:19,751 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:33:19,757 p=31718 u=zuul n=ansible | TASK [repo_setup : Create download directory path={{ cifmw_repo_setup_rhos_release_path }}, state=directory, mode=0755] *** 2026-03-09 18:33:19,757 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.027) 0:00:15.693 ********** 2026-03-09 18:33:19,757 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.027) 0:00:15.691 ********** 2026-03-09 18:33:19,767 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:19,773 p=31718 u=zuul n=ansible | TASK [repo_setup : Print the URL to request msg={{ cifmw_repo_setup_rhos_release_rpm }}] *** 2026-03-09 18:33:19,773 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.016) 0:00:15.709 ********** 2026-03-09 18:33:19,773 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.016) 0:00:15.707 ********** 2026-03-09 18:33:19,783 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:19,790 p=31718 u=zuul n=ansible | TASK [Download the RPM name=krb_request] *************************************** 2026-03-09 18:33:19,791 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.726 ********** 2026-03-09 18:33:19,791 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.017) 0:00:15.725 ********** 2026-03-09 18:33:19,804 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:19,810 p=31718 u=zuul n=ansible | TASK [repo_setup : Install RHOS Release tool name={{ cifmw_repo_setup_rhos_release_rpm if cifmw_repo_setup_rhos_release_rpm is not url else cifmw_krb_request_out.path }}, state=present, disable_gpg_check={{ cifmw_repo_setup_rhos_release_gpg_check | bool }}] *** 2026-03-09 18:33:19,810 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.019) 0:00:15.746 ********** 2026-03-09 18:33:19,810 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.019) 0:00:15.744 ********** 2026-03-09 18:33:19,819 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:19,825 p=31718 u=zuul n=ansible | TASK [repo_setup : Get rhos-release tool version _raw_params=rhos-release --version] *** 2026-03-09 18:33:19,825 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.015) 0:00:15.761 ********** 2026-03-09 18:33:19,825 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.015) 0:00:15.759 ********** 2026-03-09 18:33:19,833 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:19,839 p=31718 u=zuul n=ansible | TASK [repo_setup : Print rhos-release tool version msg={{ rr_version.stdout }}] *** 2026-03-09 18:33:19,839 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.014) 0:00:15.775 ********** 2026-03-09 18:33:19,839 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.014) 0:00:15.773 ********** 2026-03-09 18:33:19,848 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:19,855 p=31718 u=zuul n=ansible | TASK [repo_setup : Generate repos using rhos-release {{ cifmw_repo_setup_rhos_release_args }} _raw_params=rhos-release {{ cifmw_repo_setup_rhos_release_args }} \ -t {{ cifmw_repo_setup_output }}] *** 2026-03-09 18:33:19,855 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.015) 0:00:15.791 ********** 2026-03-09 18:33:19,855 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.015) 0:00:15.789 ********** 2026-03-09 18:33:19,864 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:19,870 p=31718 u=zuul n=ansible | TASK [repo_setup : Check for /etc/ci/mirror_info.sh path=/etc/ci/mirror_info.sh] *** 2026-03-09 18:33:19,870 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.014) 0:00:15.805 ********** 2026-03-09 18:33:19,870 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:19 +0000 (0:00:00.014) 0:00:15.804 ********** 2026-03-09 18:33:20,069 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:33:20,078 p=31718 u=zuul n=ansible | TASK [repo_setup : Use RDO proxy mirrors chdir={{ cifmw_repo_setup_output }}, _raw_params=set -o pipefail source /etc/ci/mirror_info.sh sed -i -e "s|https://trunk.rdoproject.org|$NODEPOOL_RDO_PROXY|g" *.repo ] *** 2026-03-09 18:33:20,078 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:20 +0000 (0:00:00.208) 0:00:16.014 ********** 2026-03-09 18:33:20,078 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:20 +0000 (0:00:00.208) 0:00:16.013 ********** 2026-03-09 18:33:20,284 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:20,293 p=31718 u=zuul n=ansible | TASK [repo_setup : Use RDO CentOS mirrors (remove CentOS 10 conditional when Nodepool mirrors exist) chdir={{ cifmw_repo_setup_output }}, _raw_params=set -o pipefail source /etc/ci/mirror_info.sh sed -i -e "s|http://mirror.stream.centos.org|$NODEPOOL_CENTOS_MIRROR|g" *.repo ] *** 2026-03-09 18:33:20,293 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:20 +0000 (0:00:00.214) 0:00:16.229 ********** 2026-03-09 18:33:20,293 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:20 +0000 (0:00:00.214) 0:00:16.227 ********** 2026-03-09 18:33:20,547 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:20,553 p=31718 u=zuul n=ansible | TASK [repo_setup : Check for gating.repo file on content provider url=http://{{ content_provider_registry_ip }}:8766/gating.repo] *** 2026-03-09 18:33:20,553 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:20 +0000 (0:00:00.259) 0:00:16.489 ********** 2026-03-09 18:33:20,553 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:20 +0000 (0:00:00.259) 0:00:16.487 ********** 2026-03-09 18:33:21,113 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:33:21,120 p=31718 u=zuul n=ansible | TASK [repo_setup : Populate gating repo from content provider ip content=[gating-repo] baseurl=http://{{ content_provider_registry_ip }}:8766/ enabled=1 gpgcheck=0 priority=1 , dest={{ cifmw_repo_setup_output }}/gating.repo, mode=0644] *** 2026-03-09 18:33:21,120 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:21 +0000 (0:00:00.566) 0:00:17.056 ********** 2026-03-09 18:33:21,120 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:21 +0000 (0:00:00.566) 0:00:17.054 ********** 2026-03-09 18:33:21,685 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:21,691 p=31718 u=zuul n=ansible | TASK [repo_setup : Check for DLRN repo at the destination path={{ cifmw_repo_setup_output }}/delorean.repo] *** 2026-03-09 18:33:21,692 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:21 +0000 (0:00:00.571) 0:00:17.627 ********** 2026-03-09 18:33:21,692 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:21 +0000 (0:00:00.571) 0:00:17.626 ********** 2026-03-09 18:33:21,912 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:33:21,920 p=31718 u=zuul n=ansible | TASK [repo_setup : Lower the priority of DLRN repos to allow installation from gating repo path={{ cifmw_repo_setup_output }}/delorean.repo, regexp=priority=1, replace=priority=20] *** 2026-03-09 18:33:21,920 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:21 +0000 (0:00:00.228) 0:00:17.856 ********** 2026-03-09 18:33:21,920 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:21 +0000 (0:00:00.228) 0:00:17.854 ********** 2026-03-09 18:33:22,245 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:22,253 p=31718 u=zuul n=ansible | TASK [repo_setup : Check for DLRN component repo path={{ cifmw_repo_setup_output }}/{{ _comp_repo }}] *** 2026-03-09 18:33:22,253 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:22 +0000 (0:00:00.333) 0:00:18.189 ********** 2026-03-09 18:33:22,253 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:22 +0000 (0:00:00.333) 0:00:18.187 ********** 2026-03-09 18:33:22,278 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:22,285 p=31718 u=zuul n=ansible | TASK [repo_setup : Lower the priority of component repos to allow installation from gating repo path={{ cifmw_repo_setup_output }}/{{ _comp_repo }}, regexp=priority=1, replace=priority=2] *** 2026-03-09 18:33:22,285 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:22 +0000 (0:00:00.031) 0:00:18.221 ********** 2026-03-09 18:33:22,285 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:22 +0000 (0:00:00.031) 0:00:18.219 ********** 2026-03-09 18:33:22,309 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:33:22,315 p=31718 u=zuul n=ansible | TASK [repo_setup : Find existing repos from /etc/yum.repos.d directory paths=/etc/yum.repos.d/, patterns=*.repo, recurse=False] *** 2026-03-09 18:33:22,316 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:22 +0000 (0:00:00.030) 0:00:18.251 ********** 2026-03-09 18:33:22,316 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:22 +0000 (0:00:00.030) 0:00:18.250 ********** 2026-03-09 18:33:22,700 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:33:22,706 p=31718 u=zuul n=ansible | TASK [repo_setup : Remove existing repos from /etc/yum.repos.d directory path={{ item }}, state=absent] *** 2026-03-09 18:33:22,706 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:22 +0000 (0:00:00.390) 0:00:18.642 ********** 2026-03-09 18:33:22,706 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:22 +0000 (0:00:00.390) 0:00:18.640 ********** 2026-03-09 18:33:22,972 p=31718 u=zuul n=ansible | changed: [controller] => (item=/etc/yum.repos.d/centos-addons.repo) 2026-03-09 18:33:23,159 p=31718 u=zuul n=ansible | changed: [controller] => (item=/etc/yum.repos.d/centos.repo) 2026-03-09 18:33:23,165 p=31718 u=zuul n=ansible | TASK [repo_setup : Cleanup existing metadata _raw_params=dnf clean metadata] *** 2026-03-09 18:33:23,166 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:23 +0000 (0:00:00.459) 0:00:19.101 ********** 2026-03-09 18:33:23,166 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:23 +0000 (0:00:00.459) 0:00:19.100 ********** 2026-03-09 18:33:23,577 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:23,585 p=31718 u=zuul n=ansible | TASK [repo_setup : Copy generated repos to /etc/yum.repos.d directory mode=0755, remote_src=True, src={{ cifmw_repo_setup_output }}/, dest=/etc/yum.repos.d] *** 2026-03-09 18:33:23,585 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:23 +0000 (0:00:00.419) 0:00:19.521 ********** 2026-03-09 18:33:23,585 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:23 +0000 (0:00:00.419) 0:00:19.519 ********** 2026-03-09 18:33:23,924 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:23,935 p=31718 u=zuul n=ansible | TASK [ci_setup : Gather variables for each operating system _raw_params={{ item }}] *** 2026-03-09 18:33:23,935 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:23 +0000 (0:00:00.350) 0:00:19.871 ********** 2026-03-09 18:33:23,935 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:23 +0000 (0:00:00.350) 0:00:19.870 ********** 2026-03-09 18:33:23,984 p=31718 u=zuul n=ansible | ok: [controller] => (item=/home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/ci_setup/vars/redhat.yml) 2026-03-09 18:33:23,990 p=31718 u=zuul n=ansible | TASK [ci_setup : List packages to install var=cifmw_ci_setup_packages] ********* 2026-03-09 18:33:23,990 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:23 +0000 (0:00:00.054) 0:00:19.926 ********** 2026-03-09 18:33:23,990 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:23 +0000 (0:00:00.054) 0:00:19.924 ********** 2026-03-09 18:33:24,007 p=31718 u=zuul n=ansible | ok: [controller] => cifmw_ci_setup_packages: - bash-completion - ca-certificates - git-core - make - tar - tmux - python3-pip 2026-03-09 18:33:24,013 p=31718 u=zuul n=ansible | TASK [ci_setup : Install needed packages name={{ cifmw_ci_setup_packages }}, state=latest] *** 2026-03-09 18:33:24,013 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:24 +0000 (0:00:00.022) 0:00:19.948 ********** 2026-03-09 18:33:24,013 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:24 +0000 (0:00:00.022) 0:00:19.947 ********** 2026-03-09 18:33:54,695 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:54,703 p=31718 u=zuul n=ansible | TASK [ci_setup : Gather version of openshift client _raw_params=oc version --client -o yaml] *** 2026-03-09 18:33:54,703 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:54 +0000 (0:00:30.690) 0:00:50.639 ********** 2026-03-09 18:33:54,704 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:54 +0000 (0:00:30.690) 0:00:50.638 ********** 2026-03-09 18:33:54,946 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:33:54,952 p=31718 u=zuul n=ansible | TASK [ci_setup : Ensure openshift client install path is present path={{ cifmw_ci_setup_oc_install_path }}, state=directory, mode=0755] *** 2026-03-09 18:33:54,953 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:54 +0000 (0:00:00.249) 0:00:50.888 ********** 2026-03-09 18:33:54,953 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:54 +0000 (0:00:00.249) 0:00:50.887 ********** 2026-03-09 18:33:55,150 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:33:55,156 p=31718 u=zuul n=ansible | TASK [ci_setup : Install openshift client src={{ cifmw_ci_setup_openshift_client_download_uri }}/{{ cifmw_ci_setup_openshift_client_version }}/openshift-client-linux.tar.gz, dest={{ cifmw_ci_setup_oc_install_path }}, remote_src=True, mode=0755, creates={{ cifmw_ci_setup_oc_install_path }}/oc] *** 2026-03-09 18:33:55,157 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:55 +0000 (0:00:00.204) 0:00:51.092 ********** 2026-03-09 18:33:55,157 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:33:55 +0000 (0:00:00.204) 0:00:51.091 ********** 2026-03-09 18:34:00,843 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:34:00,853 p=31718 u=zuul n=ansible | TASK [ci_setup : Add the OC path to cifmw_path if needed cifmw_path={{ cifmw_ci_setup_oc_install_path }}:{{ ansible_env.PATH }}, cacheable=True] *** 2026-03-09 18:34:00,853 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:00 +0000 (0:00:05.696) 0:00:56.788 ********** 2026-03-09 18:34:00,853 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:00 +0000 (0:00:05.696) 0:00:56.787 ********** 2026-03-09 18:34:00,876 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:00,883 p=31718 u=zuul n=ansible | TASK [ci_setup : Create completion file] *************************************** 2026-03-09 18:34:00,883 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:00 +0000 (0:00:00.030) 0:00:56.819 ********** 2026-03-09 18:34:00,883 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:00 +0000 (0:00:00.030) 0:00:56.818 ********** 2026-03-09 18:34:01,213 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:34:01,220 p=31718 u=zuul n=ansible | TASK [ci_setup : Source completion from within .bashrc create=True, mode=0644, path={{ ansible_user_dir }}/.bashrc, block=if [ -f ~/.oc_completion ]; then source ~/.oc_completion fi] *** 2026-03-09 18:34:01,220 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.336) 0:00:57.156 ********** 2026-03-09 18:34:01,220 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.336) 0:00:57.155 ********** 2026-03-09 18:34:01,587 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:34:01,593 p=31718 u=zuul n=ansible | TASK [ci_setup : Check rhsm status _raw_params=subscription-manager status] **** 2026-03-09 18:34:01,593 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.372) 0:00:57.529 ********** 2026-03-09 18:34:01,593 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.372) 0:00:57.527 ********** 2026-03-09 18:34:01,611 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:01,617 p=31718 u=zuul n=ansible | TASK [ci_setup : Gather the repos to be enabled _repos={{ cifmw_ci_setup_rhel_rhsm_default_repos + (cifmw_ci_setup_rhel_rhsm_extra_repos | default([])) }}] *** 2026-03-09 18:34:01,618 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.024) 0:00:57.553 ********** 2026-03-09 18:34:01,618 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.024) 0:00:57.552 ********** 2026-03-09 18:34:01,635 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:01,644 p=31718 u=zuul n=ansible | TASK [ci_setup : Enabling the required repositories. name={{ item }}, state={{ rhsm_repo_state | default('enabled') }}] *** 2026-03-09 18:34:01,644 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.026) 0:00:57.580 ********** 2026-03-09 18:34:01,644 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.026) 0:00:57.579 ********** 2026-03-09 18:34:01,660 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:01,666 p=31718 u=zuul n=ansible | TASK [ci_setup : Get current /etc/redhat-release _raw_params=cat /etc/redhat-release] *** 2026-03-09 18:34:01,667 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.022) 0:00:57.602 ********** 2026-03-09 18:34:01,667 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.022) 0:00:57.601 ********** 2026-03-09 18:34:01,681 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:01,690 p=31718 u=zuul n=ansible | TASK [ci_setup : Print current /etc/redhat-release msg={{ _current_rh_release.stdout }}] *** 2026-03-09 18:34:01,690 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.023) 0:00:57.626 ********** 2026-03-09 18:34:01,690 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.023) 0:00:57.624 ********** 2026-03-09 18:34:01,702 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:01,709 p=31718 u=zuul n=ansible | TASK [ci_setup : Ensure the repos are enabled in the system using yum name={{ item.name }}, baseurl={{ item.baseurl }}, description={{ item.description | default(item.name) }}, gpgcheck={{ item.gpgcheck | default(false) }}, enabled=True, state={{ yum_repo_state | default('present') }}] *** 2026-03-09 18:34:01,709 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.018) 0:00:57.644 ********** 2026-03-09 18:34:01,709 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.018) 0:00:57.643 ********** 2026-03-09 18:34:01,724 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:01,732 p=31718 u=zuul n=ansible | TASK [ci_setup : Manage directories path={{ item }}, state={{ directory_state }}, mode=0755, owner={{ ansible_user_id }}, group={{ ansible_user_id }}] *** 2026-03-09 18:34:01,732 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.023) 0:00:57.667 ********** 2026-03-09 18:34:01,732 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:01 +0000 (0:00:00.023) 0:00:57.666 ********** 2026-03-09 18:34:01,970 p=31718 u=zuul n=ansible | changed: [controller] => (item=/home/zuul/ci-framework-data/artifacts/manifests/openstack/cr) 2026-03-09 18:34:02,207 p=31718 u=zuul n=ansible | changed: [controller] => (item=/home/zuul/ci-framework-data/logs) 2026-03-09 18:34:02,438 p=31718 u=zuul n=ansible | ok: [controller] => (item=/home/zuul/ci-framework-data/tmp) 2026-03-09 18:34:02,665 p=31718 u=zuul n=ansible | changed: [controller] => (item=/home/zuul/ci-framework-data/volumes) 2026-03-09 18:34:02,885 p=31718 u=zuul n=ansible | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters) 2026-03-09 18:34:02,903 p=31718 u=zuul n=ansible | TASK [Prepare install_yamls make targets name=install_yamls, apply={'tags': ['bootstrap']}] *** 2026-03-09 18:34:02,904 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:02 +0000 (0:00:01.171) 0:00:58.839 ********** 2026-03-09 18:34:02,904 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:02 +0000 (0:00:01.171) 0:00:58.838 ********** 2026-03-09 18:34:03,101 p=31718 u=zuul n=ansible | TASK [install_yamls : Ensure directories exist path={{ item }}, state=directory, mode=0755] *** 2026-03-09 18:34:03,101 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:03 +0000 (0:00:00.197) 0:00:59.037 ********** 2026-03-09 18:34:03,101 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:03 +0000 (0:00:00.197) 0:00:59.036 ********** 2026-03-09 18:34:03,348 p=31718 u=zuul n=ansible | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts) 2026-03-09 18:34:03,564 p=31718 u=zuul n=ansible | changed: [controller] => (item=/home/zuul/ci-framework-data/artifacts/roles/install_yamls_makes/tasks) 2026-03-09 18:34:03,745 p=31718 u=zuul n=ansible | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters) 2026-03-09 18:34:03,752 p=31718 u=zuul n=ansible | TASK [Create variables with local repos based on Zuul items name=install_yamls, tasks_from=zuul_set_operators_repo.yml] *** 2026-03-09 18:34:03,752 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:03 +0000 (0:00:00.650) 0:00:59.688 ********** 2026-03-09 18:34:03,752 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:03 +0000 (0:00:00.650) 0:00:59.687 ********** 2026-03-09 18:34:03,787 p=31718 u=zuul n=ansible | TASK [install_yamls : Set fact with local repos based on Zuul items cifmw_install_yamls_operators_repo={{ cifmw_install_yamls_operators_repo | default({}) | combine(_repo_operator_info | items2dict) }}] *** 2026-03-09 18:34:03,787 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:03 +0000 (0:00:00.035) 0:00:59.723 ********** 2026-03-09 18:34:03,788 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:03 +0000 (0:00:00.035) 0:00:59.722 ********** 2026-03-09 18:34:03,811 p=31718 u=zuul n=ansible | skipping: [controller] => (item={'branch': 'master', 'change': '979276', 'change_url': 'https://review.opendev.org/c/openstack/watcher/+/979276', 'commit_id': '64164197e0a8ae9e5016567724b13e7a66130ae7', 'patchset': '3', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher', 'name': 'openstack/watcher', 'short_name': 'watcher', 'src_dir': 'src/opendev.org/openstack/watcher'}, 'topic': None}) 2026-03-09 18:34:03,816 p=31718 u=zuul n=ansible | skipping: [controller] => (item={'branch': 'master', 'change': '979643', 'change_url': 'https://review.opendev.org/c/openstack/watcher/+/979643', 'commit_id': '29e4137808ab2295c733b36b9851bdace3dec60a', 'patchset': '3', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher', 'name': 'openstack/watcher', 'short_name': 'watcher', 'src_dir': 'src/opendev.org/openstack/watcher'}, 'topic': None}) 2026-03-09 18:34:03,822 p=31718 u=zuul n=ansible | skipping: [controller] => (item={'branch': 'master', 'change': '958644', 'change_url': 'https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/958644', 'commit_id': '5931d8d6e69fb6fa67320bbc508b3744b12bed85', 'patchset': '32', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher-tempest-plugin', 'name': 'openstack/watcher-tempest-plugin', 'short_name': 'watcher-tempest-plugin', 'src_dir': 'src/opendev.org/openstack/watcher-tempest-plugin'}, 'topic': None}) 2026-03-09 18:34:03,828 p=31718 u=zuul n=ansible | skipping: [controller] => (item={'branch': 'master', 'change': '962702', 'change_url': 'https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/962702', 'commit_id': 'b0c5f8ac8fc6c508dc18eab2816d38f4c959083b', 'patchset': '25', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher-tempest-plugin', 'name': 'openstack/watcher-tempest-plugin', 'short_name': 'watcher-tempest-plugin', 'src_dir': 'src/opendev.org/openstack/watcher-tempest-plugin'}, 'topic': None}) 2026-03-09 18:34:03,834 p=31718 u=zuul n=ansible | skipping: [controller] => (item={'branch': 'master', 'change': '963559', 'change_url': 'https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/963559', 'commit_id': '6fedf196e61ec8ee8c9655e8fa463f714bd0ce13', 'patchset': '16', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher-tempest-plugin', 'name': 'openstack/watcher-tempest-plugin', 'short_name': 'watcher-tempest-plugin', 'src_dir': 'src/opendev.org/openstack/watcher-tempest-plugin'}, 'topic': None}) 2026-03-09 18:34:03,862 p=31718 u=zuul n=ansible | ok: [controller] => (item={'branch': 'main', 'change': '351', 'change_url': 'https://github.com/openstack-k8s-operators/watcher-operator/pull/351', 'commit_id': '2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'patchset': '2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'project': {'canonical_hostname': 'github.com', 'canonical_name': 'github.com/openstack-k8s-operators/watcher-operator', 'name': 'openstack-k8s-operators/watcher-operator', 'short_name': 'watcher-operator', 'src_dir': 'src/github.com/openstack-k8s-operators/watcher-operator'}, 'topic': None}) 2026-03-09 18:34:03,874 p=31718 u=zuul n=ansible | TASK [install_yamls : Print helpful data for debugging msg=_repo_operator_name: {{ _repo_operator_name }} _repo_operator_info: {{ _repo_operator_info }} cifmw_install_yamls_operators_repo: {{ cifmw_install_yamls_operators_repo }} ] *** 2026-03-09 18:34:03,874 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:03 +0000 (0:00:00.086) 0:00:59.810 ********** 2026-03-09 18:34:03,874 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:03 +0000 (0:00:00.086) 0:00:59.808 ********** 2026-03-09 18:34:03,897 p=31718 u=zuul n=ansible | skipping: [controller] => (item={'branch': 'master', 'change': '979276', 'change_url': 'https://review.opendev.org/c/openstack/watcher/+/979276', 'commit_id': '64164197e0a8ae9e5016567724b13e7a66130ae7', 'patchset': '3', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher', 'name': 'openstack/watcher', 'short_name': 'watcher', 'src_dir': 'src/opendev.org/openstack/watcher'}, 'topic': None}) 2026-03-09 18:34:03,904 p=31718 u=zuul n=ansible | skipping: [controller] => (item={'branch': 'master', 'change': '979643', 'change_url': 'https://review.opendev.org/c/openstack/watcher/+/979643', 'commit_id': '29e4137808ab2295c733b36b9851bdace3dec60a', 'patchset': '3', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher', 'name': 'openstack/watcher', 'short_name': 'watcher', 'src_dir': 'src/opendev.org/openstack/watcher'}, 'topic': None}) 2026-03-09 18:34:03,909 p=31718 u=zuul n=ansible | skipping: [controller] => (item={'branch': 'master', 'change': '958644', 'change_url': 'https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/958644', 'commit_id': '5931d8d6e69fb6fa67320bbc508b3744b12bed85', 'patchset': '32', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher-tempest-plugin', 'name': 'openstack/watcher-tempest-plugin', 'short_name': 'watcher-tempest-plugin', 'src_dir': 'src/opendev.org/openstack/watcher-tempest-plugin'}, 'topic': None}) 2026-03-09 18:34:03,914 p=31718 u=zuul n=ansible | skipping: [controller] => (item={'branch': 'master', 'change': '962702', 'change_url': 'https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/962702', 'commit_id': 'b0c5f8ac8fc6c508dc18eab2816d38f4c959083b', 'patchset': '25', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher-tempest-plugin', 'name': 'openstack/watcher-tempest-plugin', 'short_name': 'watcher-tempest-plugin', 'src_dir': 'src/opendev.org/openstack/watcher-tempest-plugin'}, 'topic': None}) 2026-03-09 18:34:03,920 p=31718 u=zuul n=ansible | skipping: [controller] => (item={'branch': 'master', 'change': '963559', 'change_url': 'https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/963559', 'commit_id': '6fedf196e61ec8ee8c9655e8fa463f714bd0ce13', 'patchset': '16', 'project': {'canonical_hostname': 'opendev.org', 'canonical_name': 'opendev.org/openstack/watcher-tempest-plugin', 'name': 'openstack/watcher-tempest-plugin', 'short_name': 'watcher-tempest-plugin', 'src_dir': 'src/opendev.org/openstack/watcher-tempest-plugin'}, 'topic': None}) 2026-03-09 18:34:03,950 p=31718 u=zuul n=ansible | ok: [controller] => (item={'branch': 'main', 'change': '351', 'change_url': 'https://github.com/openstack-k8s-operators/watcher-operator/pull/351', 'commit_id': '2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'patchset': '2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'project': {'canonical_hostname': 'github.com', 'canonical_name': 'github.com/openstack-k8s-operators/watcher-operator', 'name': 'openstack-k8s-operators/watcher-operator', 'short_name': 'watcher-operator', 'src_dir': 'src/github.com/openstack-k8s-operators/watcher-operator'}, 'topic': None}) => msg: | _repo_operator_name: watcher _repo_operator_info: [{'key': 'WATCHER_REPO', 'value': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator'}, {'key': 'WATCHER_BRANCH', 'value': ''}] cifmw_install_yamls_operators_repo: {'WATCHER_REPO': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator', 'WATCHER_BRANCH': ''} 2026-03-09 18:34:03,963 p=31718 u=zuul n=ansible | TASK [Customize install_yamls devsetup vars if needed name=install_yamls, tasks_from=customize_devsetup_vars.yml] *** 2026-03-09 18:34:03,963 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:03 +0000 (0:00:00.089) 0:00:59.899 ********** 2026-03-09 18:34:03,963 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:03 +0000 (0:00:00.089) 0:00:59.897 ********** 2026-03-09 18:34:04,002 p=31718 u=zuul n=ansible | TASK [install_yamls : Update opm_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^opm_version:, line=opm_version: {{ cifmw_install_yamls_opm_version }}, state=present] *** 2026-03-09 18:34:04,002 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.038) 0:00:59.938 ********** 2026-03-09 18:34:04,002 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.038) 0:00:59.936 ********** 2026-03-09 18:34:04,018 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:04,024 p=31718 u=zuul n=ansible | TASK [install_yamls : Update sdk_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^sdk_version:, line=sdk_version: {{ cifmw_install_yamls_sdk_version }}, state=present] *** 2026-03-09 18:34:04,024 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.022) 0:00:59.960 ********** 2026-03-09 18:34:04,024 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.022) 0:00:59.959 ********** 2026-03-09 18:34:04,041 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:04,047 p=31718 u=zuul n=ansible | TASK [install_yamls : Update go_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^go_version:, line=go_version: {{ cifmw_install_yamls_go_version }}, state=present] *** 2026-03-09 18:34:04,047 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.022) 0:00:59.983 ********** 2026-03-09 18:34:04,047 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.022) 0:00:59.981 ********** 2026-03-09 18:34:04,063 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:04,069 p=31718 u=zuul n=ansible | TASK [install_yamls : Update kustomize_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^kustomize_version:, line=kustomize_version: {{ cifmw_install_yamls_kustomize_version }}, state=present] *** 2026-03-09 18:34:04,069 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.021) 0:01:00.005 ********** 2026-03-09 18:34:04,069 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.021) 0:01:00.003 ********** 2026-03-09 18:34:04,086 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:04,098 p=31718 u=zuul n=ansible | TASK [install_yamls : Compute the cifmw_install_yamls_vars final value _install_yamls_override_vars={{ _install_yamls_override_vars | default({}) | combine(item, recursive=True) }}] *** 2026-03-09 18:34:04,098 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.028) 0:01:00.034 ********** 2026-03-09 18:34:04,098 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.028) 0:01:00.032 ********** 2026-03-09 18:34:04,185 p=31718 u=zuul n=ansible | ok: [controller] => (item={'BMO_SETUP': False}) 2026-03-09 18:34:04,193 p=31718 u=zuul n=ansible | TASK [install_yamls : Set environment override cifmw_install_yamls_environment fact cifmw_install_yamls_environment={{ _install_yamls_override_vars.keys() | map('upper') | zip(_install_yamls_override_vars.values()) | items2dict(key_name=0, value_name=1) | combine({ 'OUT': cifmw_install_yamls_manifests_dir, 'OUTPUT_DIR': cifmw_install_yamls_edpm_dir, 'CHECKOUT_FROM_OPENSTACK_REF': cifmw_install_yamls_checkout_openstack_ref, 'OPENSTACK_K8S_BRANCH': (zuul is defined and not zuul.branch |regex_search('master|antelope|rhos')) | ternary(zuul.branch, 'main') }) | combine(install_yamls_operators_repos) }}, cacheable=True] *** 2026-03-09 18:34:04,193 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.095) 0:01:00.129 ********** 2026-03-09 18:34:04,193 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.095) 0:01:00.128 ********** 2026-03-09 18:34:04,231 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:04,237 p=31718 u=zuul n=ansible | TASK [install_yamls : Get environment structure base_path={{ cifmw_install_yamls_repo }}] *** 2026-03-09 18:34:04,237 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.044) 0:01:00.173 ********** 2026-03-09 18:34:04,237 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.044) 0:01:00.172 ********** 2026-03-09 18:34:04,814 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:04,820 p=31718 u=zuul n=ansible | TASK [install_yamls : Ensure Output directory exists path={{ cifmw_install_yamls_out_dir }}, state=directory, mode=0755] *** 2026-03-09 18:34:04,820 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.582) 0:01:00.756 ********** 2026-03-09 18:34:04,820 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:04 +0000 (0:00:00.582) 0:01:00.755 ********** 2026-03-09 18:34:05,027 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:05,033 p=31718 u=zuul n=ansible | TASK [install_yamls : Ensure user cifmw_install_yamls_vars contains existing Makefile variables that=_cifmw_install_yamls_unmatched_vars | length == 0, msg=cifmw_install_yamls_vars contains a variable that is not defined in install_yamls Makefile nor cifmw_install_yamls_whitelisted_vars: {{ _cifmw_install_yamls_unmatched_vars | join(', ')}}, quiet=True] *** 2026-03-09 18:34:05,033 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:05 +0000 (0:00:00.213) 0:01:00.969 ********** 2026-03-09 18:34:05,034 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:05 +0000 (0:00:00.213) 0:01:00.968 ********** 2026-03-09 18:34:05,061 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:05,071 p=31718 u=zuul n=ansible | TASK [install_yamls : Generate /home/zuul/ci-framework-data/artifacts/install_yamls.sh dest={{ cifmw_install_yamls_out_dir }}/{{ cifmw_install_yamls_envfile }}, content={% for k,v in cifmw_install_yamls_environment.items() %} export {{ k }}={{ v }} {% endfor %}, mode=0644] *** 2026-03-09 18:34:05,071 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:05 +0000 (0:00:00.037) 0:01:01.007 ********** 2026-03-09 18:34:05,071 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:05 +0000 (0:00:00.037) 0:01:01.005 ********** 2026-03-09 18:34:05,609 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:34:05,616 p=31718 u=zuul n=ansible | TASK [install_yamls : Set install_yamls default values cifmw_install_yamls_defaults={{ get_makefiles_env_output.makefiles_values | combine(cifmw_install_yamls_environment) }}, cacheable=True] *** 2026-03-09 18:34:05,616 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:05 +0000 (0:00:00.544) 0:01:01.552 ********** 2026-03-09 18:34:05,616 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:05 +0000 (0:00:00.544) 0:01:01.550 ********** 2026-03-09 18:34:05,638 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:05,647 p=31718 u=zuul n=ansible | TASK [install_yamls : Show the env structure var=cifmw_install_yamls_environment] *** 2026-03-09 18:34:05,647 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:05 +0000 (0:00:00.031) 0:01:01.583 ********** 2026-03-09 18:34:05,647 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:05 +0000 (0:00:00.031) 0:01:01.582 ********** 2026-03-09 18:34:05,663 p=31718 u=zuul n=ansible | ok: [controller] => cifmw_install_yamls_environment: BMO_SETUP: false CHECKOUT_FROM_OPENSTACK_REF: 'true' OPENSTACK_K8S_BRANCH: main OUT: /home/zuul/ci-framework-data/artifacts/manifests OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm WATCHER_BRANCH: '' WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator 2026-03-09 18:34:05,670 p=31718 u=zuul n=ansible | TASK [install_yamls : Show the env structure defaults var=cifmw_install_yamls_defaults] *** 2026-03-09 18:34:05,670 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:05 +0000 (0:00:00.022) 0:01:01.606 ********** 2026-03-09 18:34:05,670 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:05 +0000 (0:00:00.022) 0:01:01.604 ********** 2026-03-09 18:34:05,698 p=31718 u=zuul n=ansible | ok: [controller] => cifmw_install_yamls_defaults: ADOPTED_EXTERNAL_NETWORK: 172.21.1.0/24 ADOPTED_INTERNALAPI_NETWORK: 172.17.1.0/24 ADOPTED_STORAGEMGMT_NETWORK: 172.20.1.0/24 ADOPTED_STORAGE_NETWORK: 172.18.1.0/24 ADOPTED_TENANT_NETWORK: 172.9.1.0/24 BARBICAN: config/samples/barbican_v1beta1_barbican.yaml BARBICAN_BRANCH: main BARBICAN_COMMIT_HASH: '' BARBICAN_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/config/samples/barbican_v1beta1_barbican.yaml BARBICAN_DEPL_IMG: unused BARBICAN_IMG: quay.io/openstack-k8s-operators/barbican-operator-index:latest BARBICAN_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/kuttl-test.yaml BARBICAN_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/test/kuttl/tests BARBICAN_KUTTL_NAMESPACE: barbican-kuttl-tests BARBICAN_REPO: https://github.com/openstack-k8s-operators/barbican-operator.git BARBICAN_SERVICE_ENABLED: 'true' BARBICAN_SIMPLE_CRYPTO_ENCRYPTION_KEY: sE**********U= BAREMETAL_BRANCH: main BAREMETAL_COMMIT_HASH: '' BAREMETAL_IMG: quay.io/openstack-k8s-operators/openstack-baremetal-operator-index:latest BAREMETAL_OS_CONTAINER_IMG: '' BAREMETAL_OS_IMG: '' BAREMETAL_OS_IMG_TYPE: '' BAREMETAL_REPO: https://github.com/openstack-k8s-operators/openstack-baremetal-operator.git BAREMETAL_TIMEOUT: 20m BASH_IMG: quay.io/openstack-k8s-operators/bash:latest BGP_ASN: '64999' BGP_LEAF_1: 100.65.4.1 BGP_LEAF_2: 100.64.4.1 BGP_OVN_ROUTING: 'false' BGP_PEER_ASN: '64999' BGP_SOURCE_IP: 172.30.4.2 BGP_SOURCE_IP6: f00d:f00d:f00d:f00d:f00d:f00d:f00d:42 BMAAS_BRIDGE_IPV4_PREFIX: 172.20.1.2/24 BMAAS_BRIDGE_IPV6_PREFIX: fd00:bbbb::2/64 BMAAS_INSTANCE_DISK_SIZE: '20' BMAAS_INSTANCE_MEMORY: '4096' BMAAS_INSTANCE_NAME_PREFIX: crc-bmaas BMAAS_INSTANCE_NET_MODEL: virtio BMAAS_INSTANCE_OS_VARIANT: centos-stream9 BMAAS_INSTANCE_VCPUS: '2' BMAAS_INSTANCE_VIRT_TYPE: kvm BMAAS_IPV4: 'true' BMAAS_IPV6: 'false' BMAAS_LIBVIRT_USER: sushyemu BMAAS_METALLB_ADDRESS_POOL: 172.20.1.64/26 BMAAS_METALLB_POOL_NAME: baremetal BMAAS_NETWORK_IPV4_PREFIX: 172.20.1.1/24 BMAAS_NETWORK_IPV6_PREFIX: fd00:bbbb::1/64 BMAAS_NETWORK_NAME: crc-bmaas BMAAS_NODE_COUNT: '1' BMAAS_OCP_INSTANCE_NAME: crc BMAAS_REDFISH_PASSWORD: password BMAAS_REDFISH_USERNAME: admin BMAAS_ROUTE_LIBVIRT_NETWORKS: crc-bmaas,crc,default BMAAS_SUSHY_EMULATOR_DRIVER: libvirt BMAAS_SUSHY_EMULATOR_IMAGE: quay.io/metal3-io/sushy-tools:latest BMAAS_SUSHY_EMULATOR_NAMESPACE: sushy-emulator BMAAS_SUSHY_EMULATOR_OS_CLIENT_CONFIG_FILE: /etc/openstack/clouds.yaml BMAAS_SUSHY_EMULATOR_OS_CLOUD: openstack BMH_NAMESPACE: openstack BMO_BRANCH: release-0.9 BMO_CLEANUP: 'true' BMO_COMMIT_HASH: '' BMO_IPA_BASEURI: https://tarballs.opendev.org/openstack/ironic-python-agent/dib BMO_IPA_BRANCH: stable/2024.1 BMO_IPA_FLAVOR: centos9 BMO_IPA_INSECURE: 'false' BMO_IRONIC_HOST: 192.168.122.10 BMO_PROVISIONING_INTERFACE: '' BMO_REPO: https://github.com/metal3-io/baremetal-operator BMO_SETUP: false BMO_SETUP_ROUTE_REPLACE: 'true' BM_CTLPLANE_INTERFACE: enp1s0 BM_INSTANCE_MEMORY: '8192' BM_INSTANCE_NAME_PREFIX: edpm-compute-baremetal BM_INSTANCE_NAME_SUFFIX: '0' BM_NETWORK_NAME: default BM_NODE_COUNT: '1' BM_ROOT_PASSWORD: '' BM_ROOT_PASSWORD_SECRET: '' CEILOMETER_CENTRAL_DEPL_IMG: unused CEILOMETER_NOTIFICATION_DEPL_IMG: unused CEPH_BRANCH: release-1.15 CEPH_CLIENT: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/toolbox.yaml CEPH_COMMON: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/common.yaml CEPH_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/cluster-test.yaml CEPH_CRDS: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/crds.yaml CEPH_IMG: quay.io/ceph/demo:latest-squid CEPH_OP: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/operator-openshift.yaml CEPH_REPO: https://github.com/rook/rook.git CERTMANAGER_TIMEOUT: 300s CHECKOUT_FROM_OPENSTACK_REF: 'true' CINDER: config/samples/cinder_v1beta1_cinder.yaml CINDERAPI_DEPL_IMG: unused CINDERBKP_DEPL_IMG: unused CINDERSCH_DEPL_IMG: unused CINDERVOL_DEPL_IMG: unused CINDER_BRANCH: main CINDER_COMMIT_HASH: '' CINDER_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/config/samples/cinder_v1beta1_cinder.yaml CINDER_IMG: quay.io/openstack-k8s-operators/cinder-operator-index:latest CINDER_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/kuttl-test.yaml CINDER_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/test/kuttl/tests CINDER_KUTTL_NAMESPACE: cinder-kuttl-tests CINDER_REPO: https://github.com/openstack-k8s-operators/cinder-operator.git CLEANUP_DIR_CMD: rm -Rf CRC_BGP_NIC_1_MAC: '52:54:00:11:11:11' CRC_BGP_NIC_2_MAC: '52:54:00:11:11:12' CRC_HTTPS_PROXY: '' CRC_HTTP_PROXY: '' CRC_STORAGE_NAMESPACE: crc-storage CRC_STORAGE_RETRIES: '3' CRC_URL: '''https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/crc/latest/crc-linux-amd64.tar.xz''' CRC_VERSION: latest DATAPLANE_ANSIBLE_SECRET: dataplane-ansible-ssh-private-key-secret DATAPLANE_ANSIBLE_USER: '' DATAPLANE_COMPUTE_IP: 192.168.122.100 DATAPLANE_CONTAINER_PREFIX: openstack DATAPLANE_CONTAINER_TAG: current-podified DATAPLANE_CUSTOM_SERVICE_RUNNER_IMG: quay.io/openstack-k8s-operators/openstack-ansibleee-runner:latest DATAPLANE_DEFAULT_GW: 192.168.122.1 DATAPLANE_EXTRA_NOVA_CONFIG_FILE: /dev/null DATAPLANE_GROWVOLS_ARGS: /=8GB /tmp=1GB /home=1GB /var=100% DATAPLANE_KUSTOMIZE_SCENARIO: preprovisioned DATAPLANE_NETWORKER_IP: 192.168.122.200 DATAPLANE_NETWORK_INTERFACE_NAME: eth0 DATAPLANE_NOVA_NFS_PATH: '' DATAPLANE_NTP_SERVER: pool.ntp.org DATAPLANE_PLAYBOOK: osp.edpm.download_cache DATAPLANE_REGISTRY_URL: quay.io/podified-antelope-centos9 DATAPLANE_RUNNER_IMG: '' DATAPLANE_SERVER_ROLE: compute DATAPLANE_SSHD_ALLOWED_RANGES: '[''192.168.122.0/24'']' DATAPLANE_TIMEOUT: 30m DATAPLANE_TLS_ENABLED: 'true' DATAPLANE_TOTAL_NETWORKER_NODES: '1' DATAPLANE_TOTAL_NODES: '1' DBSERVICE: galera DESIGNATE: config/samples/designate_v1beta1_designate.yaml DESIGNATE_BRANCH: main DESIGNATE_COMMIT_HASH: '' DESIGNATE_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/config/samples/designate_v1beta1_designate.yaml DESIGNATE_IMG: quay.io/openstack-k8s-operators/designate-operator-index:latest DESIGNATE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/kuttl-test.yaml DESIGNATE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/test/kuttl/tests DESIGNATE_KUTTL_NAMESPACE: designate-kuttl-tests DESIGNATE_REPO: https://github.com/openstack-k8s-operators/designate-operator.git DNSDATA: config/samples/network_v1beta1_dnsdata.yaml DNSDATA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_dnsdata.yaml DNSMASQ: config/samples/network_v1beta1_dnsmasq.yaml DNSMASQ_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_dnsmasq.yaml DNS_DEPL_IMG: unused DNS_DOMAIN: localdomain DOWNLOAD_TOOLS_SELECTION: all EDPM_ATTACH_EXTNET: 'true' EDPM_COMPUTE_ADDITIONAL_HOST_ROUTES: '''[]''' EDPM_COMPUTE_ADDITIONAL_NETWORKS: '''[]''' EDPM_COMPUTE_CELLS: '1' EDPM_COMPUTE_CEPH_ENABLED: 'true' EDPM_COMPUTE_CEPH_NOVA: 'true' EDPM_COMPUTE_DHCP_AGENT_ENABLED: 'true' EDPM_COMPUTE_SRIOV_ENABLED: 'true' EDPM_COMPUTE_SUFFIX: '0' EDPM_CONFIGURE_DEFAULT_ROUTE: 'true' EDPM_CONFIGURE_HUGEPAGES: 'false' EDPM_CONFIGURE_NETWORKING: 'true' EDPM_FIRSTBOOT_EXTRA: /tmp/edpm-firstboot-extra EDPM_NETWORKER_SUFFIX: '0' EDPM_TOTAL_NETWORKERS: '1' EDPM_TOTAL_NODES: '1' GALERA_REPLICAS: '' GENERATE_SSH_KEYS: 'true' GIT_CLONE_OPTS: '' GLANCE: config/samples/glance_v1beta1_glance.yaml GLANCEAPI_DEPL_IMG: unused GLANCE_BRANCH: main GLANCE_COMMIT_HASH: '' GLANCE_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/config/samples/glance_v1beta1_glance.yaml GLANCE_IMG: quay.io/openstack-k8s-operators/glance-operator-index:latest GLANCE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/kuttl-test.yaml GLANCE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/test/kuttl/tests GLANCE_KUTTL_NAMESPACE: glance-kuttl-tests GLANCE_REPO: https://github.com/openstack-k8s-operators/glance-operator.git HEAT: config/samples/heat_v1beta1_heat.yaml HEATAPI_DEPL_IMG: unused HEATCFNAPI_DEPL_IMG: unused HEATENGINE_DEPL_IMG: unused HEAT_AUTH_ENCRYPTION_KEY: 76**********f0 HEAT_BRANCH: main HEAT_COMMIT_HASH: '' HEAT_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/config/samples/heat_v1beta1_heat.yaml HEAT_IMG: quay.io/openstack-k8s-operators/heat-operator-index:latest HEAT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/kuttl-test.yaml HEAT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/test/kuttl/tests HEAT_KUTTL_NAMESPACE: heat-kuttl-tests HEAT_REPO: https://github.com/openstack-k8s-operators/heat-operator.git HEAT_SERVICE_ENABLED: 'true' HORIZON: config/samples/horizon_v1beta1_horizon.yaml HORIZON_BRANCH: main HORIZON_COMMIT_HASH: '' HORIZON_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/config/samples/horizon_v1beta1_horizon.yaml HORIZON_DEPL_IMG: unused HORIZON_IMG: quay.io/openstack-k8s-operators/horizon-operator-index:latest HORIZON_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/kuttl-test.yaml HORIZON_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/test/kuttl/tests HORIZON_KUTTL_NAMESPACE: horizon-kuttl-tests HORIZON_REPO: https://github.com/openstack-k8s-operators/horizon-operator.git INFRA_BRANCH: main INFRA_COMMIT_HASH: '' INFRA_IMG: quay.io/openstack-k8s-operators/infra-operator-index:latest INFRA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/kuttl-test.yaml INFRA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/test/kuttl/tests INFRA_KUTTL_NAMESPACE: infra-kuttl-tests INFRA_REPO: https://github.com/openstack-k8s-operators/infra-operator.git INSTALL_CERT_MANAGER: 'true' INSTALL_NMSTATE: true || false INSTALL_NNCP: true || false INTERNALAPI_HOST_ROUTES: '' IPV6_LAB_IPV4_NETWORK_IPADDRESS: 172.30.0.1/24 IPV6_LAB_IPV6_NETWORK_IPADDRESS: fd00:abcd:abcd:fc00::1/64 IPV6_LAB_LIBVIRT_STORAGE_POOL: default IPV6_LAB_MANAGE_FIREWALLD: 'true' IPV6_LAB_NAT64_HOST_IPV4: 172.30.0.2/24 IPV6_LAB_NAT64_HOST_IPV6: fd00:abcd:abcd:fc00::2/64 IPV6_LAB_NAT64_INSTANCE_NAME: nat64-router IPV6_LAB_NAT64_IPV6_NETWORK: fd00:abcd:abcd:fc00::/64 IPV6_LAB_NAT64_TAYGA_DYNAMIC_POOL: 192.168.255.0/24 IPV6_LAB_NAT64_TAYGA_IPV4: 192.168.255.1 IPV6_LAB_NAT64_TAYGA_IPV6: fd00:abcd:abcd:fc00::3 IPV6_LAB_NAT64_TAYGA_IPV6_PREFIX: fd00:abcd:abcd:fcff::/96 IPV6_LAB_NAT64_UPDATE_PACKAGES: 'false' IPV6_LAB_NETWORK_NAME: nat64 IPV6_LAB_SNO_CLUSTER_NETWORK: fd00:abcd:0::/48 IPV6_LAB_SNO_HOST_IP: fd00:abcd:abcd:fc00::11 IPV6_LAB_SNO_HOST_PREFIX: '64' IPV6_LAB_SNO_INSTANCE_NAME: sno IPV6_LAB_SNO_MACHINE_NETWORK: fd00:abcd:abcd:fc00::/64 IPV6_LAB_SNO_OCP_MIRROR_URL: https://mirror.openshift.com/pub/openshift-v4/clients/ocp IPV6_LAB_SNO_OCP_VERSION: latest-4.14 IPV6_LAB_SNO_SERVICE_NETWORK: fd00:abcd:abcd:fc03::/112 IPV6_LAB_SSH_PUB_KEY: /home/zuul/.ssh/id_rsa.pub IPV6_LAB_WORK_DIR: /home/zuul/.ipv6lab IRONIC: config/samples/ironic_v1beta1_ironic.yaml IRONICAPI_DEPL_IMG: unused IRONICCON_DEPL_IMG: unused IRONICINS_DEPL_IMG: unused IRONICNAG_DEPL_IMG: unused IRONICPXE_DEPL_IMG: unused IRONIC_BRANCH: main IRONIC_COMMIT_HASH: '' IRONIC_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/config/samples/ironic_v1beta1_ironic.yaml IRONIC_IMAGE: quay.io/metal3-io/ironic IRONIC_IMAGE_TAG: release-24.1 IRONIC_IMG: quay.io/openstack-k8s-operators/ironic-operator-index:latest IRONIC_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/kuttl-test.yaml IRONIC_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/test/kuttl/tests IRONIC_KUTTL_NAMESPACE: ironic-kuttl-tests IRONIC_REPO: https://github.com/openstack-k8s-operators/ironic-operator.git KEYSTONEAPI: config/samples/keystone_v1beta1_keystoneapi.yaml KEYSTONEAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/config/samples/keystone_v1beta1_keystoneapi.yaml KEYSTONEAPI_DEPL_IMG: unused KEYSTONE_BRANCH: main KEYSTONE_COMMIT_HASH: '' KEYSTONE_FEDERATION_CLIENT_SECRET: CO**********6f KEYSTONE_FEDERATION_CRYPTO_PASSPHRASE: openstack KEYSTONE_IMG: quay.io/openstack-k8s-operators/keystone-operator-index:latest KEYSTONE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/kuttl-test.yaml KEYSTONE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/test/kuttl/tests KEYSTONE_KUTTL_NAMESPACE: keystone-kuttl-tests KEYSTONE_REPO: https://github.com/openstack-k8s-operators/keystone-operator.git KUBEADMIN_PWD: '12345678' LIBVIRT_SECRET: libvirt-secret LOKI_DEPLOY_MODE: openshift-network LOKI_DEPLOY_NAMESPACE: netobserv LOKI_DEPLOY_SIZE: 1x.demo LOKI_NAMESPACE: openshift-operators-redhat LOKI_OPERATOR_GROUP: openshift-operators-redhat-loki LOKI_SUBSCRIPTION: loki-operator LVMS_CR: '1' MANILA: config/samples/manila_v1beta1_manila.yaml MANILAAPI_DEPL_IMG: unused MANILASCH_DEPL_IMG: unused MANILASHARE_DEPL_IMG: unused MANILA_BRANCH: main MANILA_COMMIT_HASH: '' MANILA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/config/samples/manila_v1beta1_manila.yaml MANILA_IMG: quay.io/openstack-k8s-operators/manila-operator-index:latest MANILA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/kuttl-test.yaml MANILA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/test/kuttl/tests MANILA_KUTTL_NAMESPACE: manila-kuttl-tests MANILA_REPO: https://github.com/openstack-k8s-operators/manila-operator.git MANILA_SERVICE_ENABLED: 'true' MARIADB: config/samples/mariadb_v1beta1_galera.yaml MARIADB_BRANCH: main MARIADB_CHAINSAW_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/chainsaw/config.yaml MARIADB_CHAINSAW_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/chainsaw/tests MARIADB_CHAINSAW_NAMESPACE: mariadb-chainsaw-tests MARIADB_COMMIT_HASH: '' MARIADB_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/config/samples/mariadb_v1beta1_galera.yaml MARIADB_DEPL_IMG: unused MARIADB_IMG: quay.io/openstack-k8s-operators/mariadb-operator-index:latest MARIADB_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/kuttl-test.yaml MARIADB_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/kuttl/tests MARIADB_KUTTL_NAMESPACE: mariadb-kuttl-tests MARIADB_REPO: https://github.com/openstack-k8s-operators/mariadb-operator.git MEMCACHED: config/samples/memcached_v1beta1_memcached.yaml MEMCACHED_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/memcached_v1beta1_memcached.yaml MEMCACHED_DEPL_IMG: unused METADATA_SHARED_SECRET: '12**********42' METALLB_IPV6_POOL: fd00:aaaa::80-fd00:aaaa::90 METALLB_POOL: 192.168.122.80-192.168.122.90 MICROSHIFT: '0' MIRROR_INSECURE: 'true' MIRROR_NAMESPACE: openstack-mirror MIRROR_REGISTRY_HOST: default-route-openshift-image-registry.apps-crc.testing NAMESPACE: openstack NETCONFIG: config/samples/network_v1beta1_netconfig.yaml NETCONFIG_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_netconfig.yaml NETCONFIG_DEPL_IMG: unused NETOBSERV_DEPLOY_NAMESPACE: netobserv NETOBSERV_NAMESPACE: openshift-netobserv-operator NETOBSERV_OPERATOR_GROUP: openshift-netobserv-operator-net NETOBSERV_SUBSCRIPTION: netobserv-operator NETWORK_BGP: 'false' NETWORK_DESIGNATE_ADDRESS_PREFIX: 172.28.0 NETWORK_DESIGNATE_EXT_ADDRESS_PREFIX: 172.50.0 NETWORK_INTERNALAPI_ADDRESS_PREFIX: 172.17.0 NETWORK_ISOLATION: 'true' NETWORK_ISOLATION_INSTANCE_NAME: crc NETWORK_ISOLATION_IPV4: 'true' NETWORK_ISOLATION_IPV4_ADDRESS: 172.16.1.1/24 NETWORK_ISOLATION_IPV4_NAT: 'true' NETWORK_ISOLATION_IPV6: 'false' NETWORK_ISOLATION_IPV6_ADDRESS: fd00:aaaa::1/64 NETWORK_ISOLATION_IP_ADDRESS: 192.168.122.10 NETWORK_ISOLATION_MAC: '52:54:00:11:11:10' NETWORK_ISOLATION_NETWORK_NAME: net-iso NETWORK_ISOLATION_NET_NAME: default NETWORK_ISOLATION_USE_DEFAULT_NETWORK: 'true' NETWORK_MTU: '1500' NETWORK_STORAGEMGMT_ADDRESS_PREFIX: 172.20.0 NETWORK_STORAGE_ADDRESS_PREFIX: 172.18.0 NETWORK_STORAGE_MACVLAN: '' NETWORK_TENANT_ADDRESS_PREFIX: 172.19.0 NETWORK_VLAN_START: '20' NETWORK_VLAN_STEP: '1' NEUTRONAPI: config/samples/neutron_v1beta1_neutronapi.yaml NEUTRONAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/config/samples/neutron_v1beta1_neutronapi.yaml NEUTRONAPI_DEPL_IMG: unused NEUTRON_BRANCH: main NEUTRON_COMMIT_HASH: '' NEUTRON_IMG: quay.io/openstack-k8s-operators/neutron-operator-index:latest NEUTRON_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/kuttl-test.yaml NEUTRON_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/test/kuttl/tests NEUTRON_KUTTL_NAMESPACE: neutron-kuttl-tests NEUTRON_REPO: https://github.com/openstack-k8s-operators/neutron-operator.git NFS_HOME: /home/nfs NMSTATE_NAMESPACE: openshift-nmstate NMSTATE_OPERATOR_GROUP: openshift-nmstate-tn6k8 NMSTATE_SUBSCRIPTION: kubernetes-nmstate-operator NNCP_ADDITIONAL_HOST_ROUTES: '' NNCP_BGP_1_INTERFACE: enp7s0 NNCP_BGP_1_IP_ADDRESS: 100.65.4.2 NNCP_BGP_2_INTERFACE: enp8s0 NNCP_BGP_2_IP_ADDRESS: 100.64.4.2 NNCP_BRIDGE: ospbr NNCP_CLEANUP_TIMEOUT: 120s NNCP_CTLPLANE_IPV6_ADDRESS_PREFIX: 'fd00:aaaa::' NNCP_CTLPLANE_IPV6_ADDRESS_SUFFIX: '10' NNCP_CTLPLANE_IP_ADDRESS_PREFIX: 192.168.122 NNCP_CTLPLANE_IP_ADDRESS_SUFFIX: '10' NNCP_DNS_SERVER: 192.168.122.1 NNCP_DNS_SERVER_IPV6: fd00:aaaa::1 NNCP_GATEWAY: 192.168.122.1 NNCP_GATEWAY_IPV6: fd00:aaaa::1 NNCP_INTERFACE: enp6s0 NNCP_NODES: '' NNCP_TIMEOUT: 240s NOVA: config/samples/nova_v1beta1_nova_collapsed_cell.yaml NOVA_BRANCH: main NOVA_COMMIT_HASH: '' NOVA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/nova-operator/config/samples/nova_v1beta1_nova_collapsed_cell.yaml NOVA_IMG: quay.io/openstack-k8s-operators/nova-operator-index:latest NOVA_REPO: https://github.com/openstack-k8s-operators/nova-operator.git NUMBER_OF_INSTANCES: '1' OCP_NETWORK_NAME: crc OCTAVIA: config/samples/octavia_v1beta1_octavia.yaml OCTAVIA_BRANCH: main OCTAVIA_COMMIT_HASH: '' OCTAVIA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/config/samples/octavia_v1beta1_octavia.yaml OCTAVIA_IMG: quay.io/openstack-k8s-operators/octavia-operator-index:latest OCTAVIA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/kuttl-test.yaml OCTAVIA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/test/kuttl/tests OCTAVIA_KUTTL_NAMESPACE: octavia-kuttl-tests OCTAVIA_REPO: https://github.com/openstack-k8s-operators/octavia-operator.git OKD: 'false' OPENSTACK_BRANCH: main OPENSTACK_BUNDLE_IMG: quay.io/openstack-k8s-operators/openstack-operator-bundle:latest OPENSTACK_COMMIT_HASH: '' OPENSTACK_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/config/samples/core_v1beta1_openstackcontrolplane_galera_network_isolation.yaml OPENSTACK_CRDS_DIR: openstack_crds OPENSTACK_CTLPLANE: config/samples/core_v1beta1_openstackcontrolplane_galera_network_isolation.yaml OPENSTACK_IMG: quay.io/openstack-k8s-operators/openstack-operator-index:latest OPENSTACK_K8S_BRANCH: main OPENSTACK_K8S_TAG: latest OPENSTACK_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/kuttl-test.yaml OPENSTACK_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/test/kuttl/tests OPENSTACK_KUTTL_NAMESPACE: openstack-kuttl-tests OPENSTACK_NEUTRON_CUSTOM_CONF: '' OPENSTACK_REPO: https://github.com/openstack-k8s-operators/openstack-operator.git OPENSTACK_STORAGE_BUNDLE_IMG: quay.io/openstack-k8s-operators/openstack-operator-storage-bundle:latest OPERATOR_BASE_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator OPERATOR_CHANNEL: '' OPERATOR_NAMESPACE: openstack-operators OPERATOR_SOURCE: '' OPERATOR_SOURCE_NAMESPACE: '' OUT: /home/zuul/ci-framework-data/artifacts/manifests OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm OVNCONTROLLER: config/samples/ovn_v1beta1_ovncontroller.yaml OVNCONTROLLER_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovncontroller.yaml OVNCONTROLLER_NMAP: 'true' OVNDBS: config/samples/ovn_v1beta1_ovndbcluster.yaml OVNDBS_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovndbcluster.yaml OVNNORTHD: config/samples/ovn_v1beta1_ovnnorthd.yaml OVNNORTHD_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovnnorthd.yaml OVN_BRANCH: main OVN_COMMIT_HASH: '' OVN_IMG: quay.io/openstack-k8s-operators/ovn-operator-index:latest OVN_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/kuttl-test.yaml OVN_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/test/kuttl/tests OVN_KUTTL_NAMESPACE: ovn-kuttl-tests OVN_REPO: https://github.com/openstack-k8s-operators/ovn-operator.git PASSWORD: '12**********78' PLACEMENTAPI: config/samples/placement_v1beta1_placementapi.yaml PLACEMENTAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/config/samples/placement_v1beta1_placementapi.yaml PLACEMENTAPI_DEPL_IMG: unused PLACEMENT_BRANCH: main PLACEMENT_COMMIT_HASH: '' PLACEMENT_IMG: quay.io/openstack-k8s-operators/placement-operator-index:latest PLACEMENT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/kuttl-test.yaml PLACEMENT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/test/kuttl/tests PLACEMENT_KUTTL_NAMESPACE: placement-kuttl-tests PLACEMENT_REPO: https://github.com/openstack-k8s-operators/placement-operator.git PULL_SECRET: /home/zuul/pull-secret.txt RABBITMQ: docs/examples/default-security-context/rabbitmq.yaml RABBITMQ_BRANCH: patches RABBITMQ_COMMIT_HASH: '' RABBITMQ_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/rabbitmq-operator/docs/examples/default-security-context/rabbitmq.yaml RABBITMQ_DEPL_IMG: unused RABBITMQ_IMG: quay.io/openstack-k8s-operators/rabbitmq-cluster-operator-index:latest RABBITMQ_REPO: https://github.com/openstack-k8s-operators/rabbitmq-cluster-operator.git REDHAT_OPERATORS: 'false' REDIS: config/samples/redis_v1beta1_redis.yaml REDIS_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator-redis/config/samples/redis_v1beta1_redis.yaml REDIS_DEPL_IMG: unused RH_REGISTRY_PWD: '' RH_REGISTRY_USER: '' SECRET: os**********et SG_CORE_DEPL_IMG: unused STANDALONE_COMPUTE_DRIVER: libvirt STANDALONE_EXTERNAL_NET_PREFFIX: 172.21.0 STANDALONE_INTERNALAPI_NET_PREFIX: 172.17.0 STANDALONE_STORAGEMGMT_NET_PREFIX: 172.20.0 STANDALONE_STORAGE_NET_PREFIX: 172.18.0 STANDALONE_TENANT_NET_PREFIX: 172.19.0 STORAGEMGMT_HOST_ROUTES: '' STORAGE_CLASS: local-storage STORAGE_HOST_ROUTES: '' SWIFT: config/samples/swift_v1beta1_swift.yaml SWIFT_BRANCH: main SWIFT_COMMIT_HASH: '' SWIFT_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/config/samples/swift_v1beta1_swift.yaml SWIFT_IMG: quay.io/openstack-k8s-operators/swift-operator-index:latest SWIFT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/kuttl-test.yaml SWIFT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/test/kuttl/tests SWIFT_KUTTL_NAMESPACE: swift-kuttl-tests SWIFT_REPO: https://github.com/openstack-k8s-operators/swift-operator.git TELEMETRY: config/samples/telemetry_v1beta1_telemetry.yaml TELEMETRY_BRANCH: main TELEMETRY_COMMIT_HASH: '' TELEMETRY_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/config/samples/telemetry_v1beta1_telemetry.yaml TELEMETRY_IMG: quay.io/openstack-k8s-operators/telemetry-operator-index:latest TELEMETRY_KUTTL_BASEDIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator TELEMETRY_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/kuttl-test.yaml TELEMETRY_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/test/kuttl/suites TELEMETRY_KUTTL_NAMESPACE: telemetry-kuttl-tests TELEMETRY_KUTTL_RELPATH: test/kuttl/suites TELEMETRY_REPO: https://github.com/openstack-k8s-operators/telemetry-operator.git TENANT_HOST_ROUTES: '' TIMEOUT: 300s TLS_ENABLED: 'false' WATCHER_BRANCH: '' WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator tripleo_deploy: 'export REGISTRY_USER:' 2026-03-09 18:34:05,705 p=31718 u=zuul n=ansible | TASK [install_yamls : Generate make targets install_yamls_path={{ cifmw_install_yamls_repo }}, output_directory={{ cifmw_install_yamls_tasks_out }}] *** 2026-03-09 18:34:05,705 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:05 +0000 (0:00:00.035) 0:01:01.641 ********** 2026-03-09 18:34:05,706 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:05 +0000 (0:00:00.035) 0:01:01.640 ********** 2026-03-09 18:34:06,034 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:06,044 p=31718 u=zuul n=ansible | TASK [install_yamls : Debug generate_make module var=cifmw_generate_makes] ***** 2026-03-09 18:34:06,044 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:06 +0000 (0:00:00.338) 0:01:01.980 ********** 2026-03-09 18:34:06,044 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:06 +0000 (0:00:00.338) 0:01:01.978 ********** 2026-03-09 18:34:06,063 p=31718 u=zuul n=ansible | ok: [controller] => cifmw_generate_makes: changed: false debug: /home/zuul/src/github.com/openstack-k8s-operators/install_yamls/Makefile: - all - help - cleanup - deploy_cleanup - wait - crc_storage - crc_storage_cleanup - crc_storage_release - crc_storage_with_retries - crc_storage_cleanup_with_retries - operator_namespace - namespace - namespace_cleanup - input - input_cleanup - crc_bmo_setup - crc_bmo_cleanup - mirror_registry - mirror_registry_secure - mirror_registry_ca - mirror_registry_cleanup - openstack_prep - openstack - openstack_wait - openstack_init - openstack_cleanup - openstack_repo - openstack_deploy_prep - openstack_deploy - openstack_wait_deploy - openstack_deploy_cleanup - openstack_update_run - update_services - update_system - openstack_patch_version - edpm_deploy_generate_keys - edpm_patch_ansible_runner_image - edpm_deploy_prep - edpm_deploy_cleanup - edpm_deploy - edpm_deploy_baremetal_prep - edpm_deploy_baremetal - edpm_wait_deploy_baremetal - edpm_wait_deploy - edpm_register_dns - edpm_nova_discover_hosts - openstack_crds - openstack_crds_cleanup - edpm_deploy_networker_prep - edpm_deploy_networker_cleanup - edpm_deploy_networker - infra_prep - infra - infra_cleanup - dns_deploy_prep - dns_deploy - dns_deploy_cleanup - netconfig_deploy_prep - netconfig_deploy - netconfig_deploy_cleanup - memcached_deploy_prep - memcached_deploy - memcached_deploy_cleanup - keystone_prep - keystone - keystone_cleanup - keystone_deploy_prep - keystone_deploy - keystone_deploy_cleanup - barbican_prep - barbican - barbican_cleanup - barbican_deploy_prep - barbican_deploy - barbican_deploy_validate - barbican_deploy_cleanup - mariadb - mariadb_cleanup - mariadb_deploy_prep - mariadb_deploy - mariadb_deploy_cleanup - placement_prep - placement - placement_cleanup - placement_deploy_prep - placement_deploy - placement_deploy_cleanup - glance_prep - glance - glance_cleanup - glance_deploy_prep - glance_deploy - glance_deploy_cleanup - ovn_prep - ovn - ovn_cleanup - ovn_deploy_prep - ovn_deploy - ovn_deploy_cleanup - neutron_prep - neutron - neutron_cleanup - neutron_deploy_prep - neutron_deploy - neutron_deploy_cleanup - cinder_prep - cinder - cinder_cleanup - cinder_deploy_prep - cinder_deploy - cinder_deploy_cleanup - rabbitmq_prep - rabbitmq - rabbitmq_cleanup - rabbitmq_deploy_prep - rabbitmq_deploy - rabbitmq_deploy_cleanup - ironic_prep - ironic - ironic_cleanup - ironic_deploy_prep - ironic_deploy - ironic_deploy_cleanup - octavia_prep - octavia - octavia_cleanup - octavia_deploy_prep - octavia_deploy - octavia_deploy_cleanup - designate_prep - designate - designate_cleanup - designate_deploy_prep - designate_deploy - designate_deploy_cleanup - nova_prep - nova - nova_cleanup - nova_deploy_prep - nova_deploy - nova_deploy_cleanup - mariadb_kuttl_run - mariadb_kuttl - kuttl_db_prep - kuttl_db_cleanup - kuttl_common_prep - kuttl_common_cleanup - keystone_kuttl_run - keystone_kuttl - barbican_kuttl_run - barbican_kuttl - placement_kuttl_run - placement_kuttl - cinder_kuttl_run - cinder_kuttl - neutron_kuttl_run - neutron_kuttl - octavia_kuttl_run - octavia_kuttl - designate_kuttl - designate_kuttl_run - ovn_kuttl_run - ovn_kuttl - infra_kuttl_run - infra_kuttl - ironic_kuttl_run - ironic_kuttl - ironic_kuttl_crc - heat_kuttl_run - heat_kuttl - heat_kuttl_crc - glance_kuttl_run - glance_kuttl - manila_kuttl_run - manila_kuttl - swift_kuttl_run - swift_kuttl - horizon_kuttl_run - horizon_kuttl - openstack_kuttl_prep - openstack_kuttl_run - openstack_kuttl_cleanup - openstack_kuttl - mariadb_chainsaw_run - mariadb_chainsaw - horizon_prep - horizon - horizon_cleanup - horizon_deploy_prep - horizon_deploy - horizon_deploy_cleanup - heat_prep - heat - heat_cleanup - heat_deploy_prep - heat_deploy - heat_deploy_cleanup - baremetal_prep - baremetal - baremetal_cleanup - ceph_help - ceph - ceph_cleanup - rook_prep - rook - rook_deploy_prep - rook_deploy - rook_crc_disk - rook_cleanup - lvms - nmstate - nncp - nncp_cleanup - netattach - netattach_cleanup - metallb - metallb_config - metallb_config_cleanup - metallb_cleanup - loki - loki_cleanup - loki_deploy - loki_deploy_cleanup - netobserv - netobserv_cleanup - netobserv_deploy - netobserv_deploy_cleanup - manila_prep - manila - manila_cleanup - manila_deploy_prep - manila_deploy - manila_deploy_cleanup - telemetry_prep - telemetry - telemetry_cleanup - telemetry_deploy_prep - telemetry_deploy - telemetry_deploy_cleanup - telemetry_kuttl_run - telemetry_kuttl - swift_prep - swift - swift_cleanup - swift_deploy_prep - swift_deploy - swift_deploy_cleanup - certmanager - certmanager_cleanup - validate_marketplace - redis_deploy_prep - redis_deploy - redis_deploy_cleanup - set_slower_etcd_profile /home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup/Makefile: - help - download_tools - nfs - nfs_cleanup - crc - crc_cleanup - crc_scrub - crc_attach_default_interface - crc_attach_default_interface_cleanup - ipv6_lab_network - ipv6_lab_network_cleanup - ipv6_lab_nat64_router - ipv6_lab_nat64_router_cleanup - ipv6_lab_sno - ipv6_lab_sno_cleanup - ipv6_lab - ipv6_lab_cleanup - attach_default_interface - attach_default_interface_cleanup - network_isolation_bridge - network_isolation_bridge_cleanup - edpm_baremetal_compute - edpm_compute - edpm_compute_bootc - edpm_ansible_runner - edpm_computes_bgp - edpm_compute_repos - edpm_compute_cleanup - edpm_networker - edpm_networker_cleanup - edpm_deploy_instance - tripleo_deploy - standalone_deploy - standalone_sync - standalone - standalone_cleanup - standalone_snapshot - standalone_revert - cifmw_prepare - cifmw_cleanup - bmaas_network - bmaas_network_cleanup - bmaas_route_crc_and_crc_bmaas_networks - bmaas_route_crc_and_crc_bmaas_networks_cleanup - bmaas_crc_attach_network - bmaas_crc_attach_network_cleanup - bmaas_crc_baremetal_bridge - bmaas_crc_baremetal_bridge_cleanup - bmaas_baremetal_net_nad - bmaas_baremetal_net_nad_cleanup - bmaas_metallb - bmaas_metallb_cleanup - bmaas_virtual_bms - bmaas_virtual_bms_cleanup - bmaas_sushy_emulator - bmaas_sushy_emulator_cleanup - bmaas_sushy_emulator_wait - bmaas_generate_nodes_yaml - bmaas - bmaas_cleanup failed: false success: true 2026-03-09 18:34:06,072 p=31718 u=zuul n=ansible | TASK [install_yamls : Create the install_yamls parameters file dest={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml, content={{ { 'cifmw_install_yamls_environment': cifmw_install_yamls_environment, 'cifmw_install_yamls_defaults': cifmw_install_yamls_defaults } | to_nice_yaml }}, mode=0644] *** 2026-03-09 18:34:06,072 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:06 +0000 (0:00:00.028) 0:01:02.008 ********** 2026-03-09 18:34:06,072 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:06 +0000 (0:00:00.028) 0:01:02.006 ********** 2026-03-09 18:34:07,000 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:34:07,007 p=31718 u=zuul n=ansible | TASK [install_yamls : Create empty cifmw_install_yamls_environment if needed cifmw_install_yamls_environment={}] *** 2026-03-09 18:34:07,007 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:07 +0000 (0:00:00.935) 0:01:02.943 ********** 2026-03-09 18:34:07,007 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:07 +0000 (0:00:00.935) 0:01:02.941 ********** 2026-03-09 18:34:07,022 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:07,036 p=31718 u=zuul n=ansible | TASK [discover_latest_image : Get latest image url={{ cifmw_discover_latest_image_base_url }}, image_prefix={{ cifmw_discover_latest_image_qcow_prefix }}, images_file={{ cifmw_discover_latest_image_images_file }}] *** 2026-03-09 18:34:07,036 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:07 +0000 (0:00:00.029) 0:01:02.972 ********** 2026-03-09 18:34:07,036 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:07 +0000 (0:00:00.029) 0:01:02.971 ********** 2026-03-09 18:34:07,398 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:34:07,405 p=31718 u=zuul n=ansible | TASK [discover_latest_image : Export facts accordingly cifmw_discovered_image_name={{ discovered_image['data']['image_name'] }}, cifmw_discovered_image_url={{ discovered_image['data']['image_url'] }}, cifmw_discovered_hash={{ discovered_image['data']['hash'] }}, cifmw_discovered_hash_algorithm={{ discovered_image['data']['hash_algorithm'] }}, cacheable=True] *** 2026-03-09 18:34:07,405 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:07 +0000 (0:00:00.369) 0:01:03.341 ********** 2026-03-09 18:34:07,406 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:07 +0000 (0:00:00.369) 0:01:03.340 ********** 2026-03-09 18:34:07,423 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:07,435 p=31718 u=zuul n=ansible | TASK [cifmw_setup : Create artifacts with custom params mode=0644, dest={{ cifmw_basedir }}/artifacts/parameters/custom-params.yml, content={{ ci_framework_params | to_nice_yaml }}] *** 2026-03-09 18:34:07,435 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:07 +0000 (0:00:00.029) 0:01:03.371 ********** 2026-03-09 18:34:07,435 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:07 +0000 (0:00:00.029) 0:01:03.369 ********** 2026-03-09 18:34:08,049 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:34:08,084 p=31718 u=zuul n=ansible | PLAY [Install dev tools] ******************************************************* 2026-03-09 18:34:08,104 p=31718 u=zuul n=ansible | TASK [Assert that operator_name is set that=['operator_name is defined']] ****** 2026-03-09 18:34:08,104 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:08 +0000 (0:00:00.668) 0:01:04.040 ********** 2026-03-09 18:34:08,104 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:08 +0000 (0:00:00.668) 0:01:04.038 ********** 2026-03-09 18:34:08,126 p=31718 u=zuul n=ansible | ok: [controller] => changed: false msg: All assertions passed 2026-03-09 18:34:08,134 p=31718 u=zuul n=ansible | TASK [Download install_yamls deps name=install_yamls_makes, tasks_from=make_download_tools] *** 2026-03-09 18:34:08,134 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:08 +0000 (0:00:00.030) 0:01:04.070 ********** 2026-03-09 18:34:08,134 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:08 +0000 (0:00:00.030) 0:01:04.069 ********** 2026-03-09 18:34:08,168 p=31718 u=zuul n=ansible | TASK [install_yamls_makes : Debug make_download_tools_env var=make_download_tools_env] *** 2026-03-09 18:34:08,168 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:08 +0000 (0:00:00.033) 0:01:04.104 ********** 2026-03-09 18:34:08,168 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:08 +0000 (0:00:00.033) 0:01:04.103 ********** 2026-03-09 18:34:08,183 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:08,191 p=31718 u=zuul n=ansible | TASK [install_yamls_makes : Debug make_download_tools_params var=make_download_tools_params] *** 2026-03-09 18:34:08,192 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:08 +0000 (0:00:00.023) 0:01:04.127 ********** 2026-03-09 18:34:08,192 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:08 +0000 (0:00:00.023) 0:01:04.126 ********** 2026-03-09 18:34:08,206 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:08,216 p=31718 u=zuul n=ansible | TASK [install_yamls_makes : Run download_tools output_dir={{ cifmw_basedir }}/artifacts, chdir=/home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup, script=make download_tools, dry_run={{ make_download_tools_dryrun|default(false)|bool }}, extra_args={{ dict((make_download_tools_env|default({})), **(make_download_tools_params|default({}))) }}] *** 2026-03-09 18:34:08,216 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:08 +0000 (0:00:00.024) 0:01:04.152 ********** 2026-03-09 18:34:08,216 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:08 +0000 (0:00:00.024) 0:01:04.151 ********** 2026-03-09 18:34:08,320 p=31718 u=zuul n=ansible | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_000_run_download.log 2026-03-09 18:34:49,145 p=31718 u=zuul n=ansible | [WARNING]: conditional statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ make_download_tools_until | default(true) }} 2026-03-09 18:34:49,146 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:34:49,157 p=31718 u=zuul n=ansible | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] *** 2026-03-09 18:34:49,158 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:40.941) 0:01:45.093 ********** 2026-03-09 18:34:49,158 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:40.941) 0:01:45.092 ********** 2026-03-09 18:34:49,218 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:49,225 p=31718 u=zuul n=ansible | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] *** 2026-03-09 18:34:49,225 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.067) 0:01:45.161 ********** 2026-03-09 18:34:49,225 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.067) 0:01:45.160 ********** 2026-03-09 18:34:49,321 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:49,328 p=31718 u=zuul n=ansible | TASK [run_hook : Loop on hooks for pre_infra _raw_params={{ hook.type }}.yml] *** 2026-03-09 18:34:49,328 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.103) 0:01:45.264 ********** 2026-03-09 18:34:49,329 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.103) 0:01:45.263 ********** 2026-03-09 18:34:49,445 p=31718 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/run_hook/tasks/playbook.yml for controller => (item={'name': 'Download needed tools', 'inventory': 'localhost,', 'connection': 'local', 'type': 'playbook', 'source': '/home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup/download_tools.yaml'}) 2026-03-09 18:34:49,454 p=31718 u=zuul n=ansible | TASK [run_hook : Set playbook path for Download needed tools cifmw_basedir={{ _bdir }}, hook_name={{ _hook_name }}, playbook_path={{ _play | realpath }}, log_path={{ _bdir }}/logs/{{ step }}_{{ _hook_name }}.log, extra_vars=-e namespace={{ cifmw_openstack_namespace }} {%- if hook.extra_vars is defined and hook.extra_vars|length > 0 -%} {% for key,value in hook.extra_vars.items() -%} {%- if key == 'file' %} -e "@{{ value }}" {%- else %} -e "{{ key }}={{ value }}" {%- endif %} {%- endfor %} {%- endif %}] *** 2026-03-09 18:34:49,454 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.125) 0:01:45.389 ********** 2026-03-09 18:34:49,454 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.125) 0:01:45.388 ********** 2026-03-09 18:34:49,489 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:49,496 p=31718 u=zuul n=ansible | TASK [run_hook : Get file stat path={{ playbook_path }}] *********************** 2026-03-09 18:34:49,496 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.041) 0:01:45.431 ********** 2026-03-09 18:34:49,496 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.041) 0:01:45.430 ********** 2026-03-09 18:34:49,697 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:49,704 p=31718 u=zuul n=ansible | TASK [run_hook : Fail if playbook doesn't exist msg=Playbook {{ playbook_path }} doesn't seem to exist.] *** 2026-03-09 18:34:49,705 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.208) 0:01:45.640 ********** 2026-03-09 18:34:49,705 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.208) 0:01:45.639 ********** 2026-03-09 18:34:49,718 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:34:49,725 p=31718 u=zuul n=ansible | TASK [run_hook : Get parameters files paths={{ (cifmw_basedir, 'artifacts/parameters') | path_join }}, file_type=file, patterns=*.yml] *** 2026-03-09 18:34:49,725 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.020) 0:01:45.661 ********** 2026-03-09 18:34:49,725 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.020) 0:01:45.660 ********** 2026-03-09 18:34:49,925 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:49,933 p=31718 u=zuul n=ansible | TASK [run_hook : Add parameters artifacts as extra variables extra_vars={{ extra_vars }} {% for file in cifmw_run_hook_parameters_files.files %} -e "@{{ file.path }}" {%- endfor %}] *** 2026-03-09 18:34:49,933 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.208) 0:01:45.869 ********** 2026-03-09 18:34:49,933 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.208) 0:01:45.868 ********** 2026-03-09 18:34:49,955 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:49,965 p=31718 u=zuul n=ansible | TASK [run_hook : Ensure log directory exists path={{ log_path | dirname }}, state=directory, mode=0755] *** 2026-03-09 18:34:49,965 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.031) 0:01:45.901 ********** 2026-03-09 18:34:49,965 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:49 +0000 (0:00:00.031) 0:01:45.899 ********** 2026-03-09 18:34:50,188 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:50,197 p=31718 u=zuul n=ansible | TASK [run_hook : Ensure artifacts directory exists path={{ cifmw_basedir }}/artifacts, state=directory, mode=0755] *** 2026-03-09 18:34:50,198 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:50 +0000 (0:00:00.232) 0:01:46.133 ********** 2026-03-09 18:34:50,198 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:50 +0000 (0:00:00.232) 0:01:46.132 ********** 2026-03-09 18:34:50,400 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:34:50,408 p=31718 u=zuul n=ansible | TASK [run_hook : Run hook without retry - Download needed tools] *************** 2026-03-09 18:34:50,408 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:50 +0000 (0:00:00.210) 0:01:46.344 ********** 2026-03-09 18:34:50,408 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:34:50 +0000 (0:00:00.210) 0:01:46.342 ********** 2026-03-09 18:34:50,535 p=31718 u=zuul n=ansible | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_001_run_hook_without_retry.log 2026-03-09 18:35:26,179 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:35:26,191 p=31718 u=zuul n=ansible | TASK [run_hook : Run hook with retry - Download needed tools] ****************** 2026-03-09 18:35:26,192 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:35.783) 0:02:22.127 ********** 2026-03-09 18:35:26,192 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:35.783) 0:02:22.126 ********** 2026-03-09 18:35:26,206 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:26,216 p=31718 u=zuul n=ansible | TASK [run_hook : Check if we have a file path={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] *** 2026-03-09 18:35:26,216 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.024) 0:02:22.152 ********** 2026-03-09 18:35:26,216 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.024) 0:02:22.151 ********** 2026-03-09 18:35:26,406 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:26,415 p=31718 u=zuul n=ansible | TASK [run_hook : Load generated content in main playbook file={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] *** 2026-03-09 18:35:26,415 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.199) 0:02:22.351 ********** 2026-03-09 18:35:26,416 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.199) 0:02:22.350 ********** 2026-03-09 18:35:26,426 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:26,461 p=31718 u=zuul n=ansible | PLAY [Prepare host virtualization] ********************************************* 2026-03-09 18:35:26,479 p=31718 u=zuul n=ansible | TASK [cifmw_setup : Load parameters files dir={{ cifmw_basedir }}/artifacts/parameters] *** 2026-03-09 18:35:26,479 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.063) 0:02:22.415 ********** 2026-03-09 18:35:26,479 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.063) 0:02:22.414 ********** 2026-03-09 18:35:26,516 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:26,524 p=31718 u=zuul n=ansible | TASK [Ensure libvirt is present/configured name=libvirt_manager] *************** 2026-03-09 18:35:26,524 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.044) 0:02:22.460 ********** 2026-03-09 18:35:26,524 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.044) 0:02:22.458 ********** 2026-03-09 18:35:26,540 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:26,547 p=31718 u=zuul n=ansible | TASK [Perpare OpenShift provisioner node name=openshift_provisioner_node] ****** 2026-03-09 18:35:26,547 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.023) 0:02:22.483 ********** 2026-03-09 18:35:26,547 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.023) 0:02:22.482 ********** 2026-03-09 18:35:26,561 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:26,598 p=31718 u=zuul n=ansible | PLAY [Build dataset hook] ****************************************************** 2026-03-09 18:35:26,624 p=31718 u=zuul n=ansible | TASK [cifmw_setup : Load parameters files dir={{ cifmw_basedir }}/artifacts/parameters] *** 2026-03-09 18:35:26,624 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.076) 0:02:22.560 ********** 2026-03-09 18:35:26,624 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.076) 0:02:22.558 ********** 2026-03-09 18:35:26,693 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:26,701 p=31718 u=zuul n=ansible | TASK [networking_mapper : Check for Networking Environment Definition file existence path={{ cifmw_networking_mapper_networking_env_def_path }}] *** 2026-03-09 18:35:26,701 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.077) 0:02:22.637 ********** 2026-03-09 18:35:26,701 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.077) 0:02:22.635 ********** 2026-03-09 18:35:26,888 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:26,897 p=31718 u=zuul n=ansible | TASK [networking_mapper : Check for Networking Definition file existance that=['_net_env_def_stat.stat.exists'], msg=Ensure that the Networking Environment Definition file exists in {{ cifmw_networking_mapper_networking_env_def_path }}, quiet=True] *** 2026-03-09 18:35:26,897 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.195) 0:02:22.833 ********** 2026-03-09 18:35:26,897 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.195) 0:02:22.831 ********** 2026-03-09 18:35:26,916 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:26,924 p=31718 u=zuul n=ansible | TASK [networking_mapper : Load the Networking Definition from file path={{ cifmw_networking_mapper_networking_env_def_path }}] *** 2026-03-09 18:35:26,924 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.027) 0:02:22.860 ********** 2026-03-09 18:35:26,925 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.027) 0:02:22.859 ********** 2026-03-09 18:35:26,943 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:26,949 p=31718 u=zuul n=ansible | TASK [networking_mapper : Set cifmw_networking_env_definition is present cifmw_networking_env_definition={{ _net_env_def_slurp['content'] | b64decode | from_yaml }}, cacheable=True] *** 2026-03-09 18:35:26,950 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.025) 0:02:22.885 ********** 2026-03-09 18:35:26,950 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.025) 0:02:22.884 ********** 2026-03-09 18:35:26,966 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:26,978 p=31718 u=zuul n=ansible | TASK [Deploy OCP using Hive name=hive] ***************************************** 2026-03-09 18:35:26,978 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.028) 0:02:22.914 ********** 2026-03-09 18:35:26,978 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:26 +0000 (0:00:00.028) 0:02:22.912 ********** 2026-03-09 18:35:26,993 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:27,001 p=31718 u=zuul n=ansible | TASK [Prepare CRC name=rhol_crc] *********************************************** 2026-03-09 18:35:27,001 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.022) 0:02:22.937 ********** 2026-03-09 18:35:27,001 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.022) 0:02:22.935 ********** 2026-03-09 18:35:27,019 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:27,025 p=31718 u=zuul n=ansible | TASK [Deploy OpenShift cluster using dev-scripts name=devscripts] ************** 2026-03-09 18:35:27,025 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.024) 0:02:22.961 ********** 2026-03-09 18:35:27,025 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.024) 0:02:22.960 ********** 2026-03-09 18:35:27,041 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:27,048 p=31718 u=zuul n=ansible | TASK [openshift_login : Ensure output directory exists path={{ cifmw_openshift_login_basedir }}/artifacts, state=directory, mode=0755] *** 2026-03-09 18:35:27,049 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.023) 0:02:22.984 ********** 2026-03-09 18:35:27,049 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.023) 0:02:22.983 ********** 2026-03-09 18:35:27,249 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:27,257 p=31718 u=zuul n=ansible | TASK [openshift_login : OpenShift login _raw_params=login.yml] ***************** 2026-03-09 18:35:27,257 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.208) 0:02:23.192 ********** 2026-03-09 18:35:27,257 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.208) 0:02:23.191 ********** 2026-03-09 18:35:27,282 p=31718 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/openshift_login/tasks/login.yml for controller 2026-03-09 18:35:27,294 p=31718 u=zuul n=ansible | TASK [openshift_login : Check if the password file is present path={{ cifmw_openshift_login_password_file | default(cifmw_openshift_password_file) }}] *** 2026-03-09 18:35:27,294 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.037) 0:02:23.230 ********** 2026-03-09 18:35:27,294 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.037) 0:02:23.229 ********** 2026-03-09 18:35:27,312 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:27,319 p=31718 u=zuul n=ansible | TASK [openshift_login : Fetch user password content src={{ cifmw_openshift_login_password_file | default(cifmw_openshift_password_file) }}] *** 2026-03-09 18:35:27,319 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.025) 0:02:23.255 ********** 2026-03-09 18:35:27,319 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.025) 0:02:23.254 ********** 2026-03-09 18:35:27,338 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:27,345 p=31718 u=zuul n=ansible | TASK [openshift_login : Set user password as a fact cifmw_openshift_login_password={{ cifmw_openshift_login_password_file_slurp.content | b64decode }}, cacheable=True] *** 2026-03-09 18:35:27,345 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.025) 0:02:23.281 ********** 2026-03-09 18:35:27,345 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.025) 0:02:23.279 ********** 2026-03-09 18:35:27,363 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:27,371 p=31718 u=zuul n=ansible | TASK [openshift_login : Set role variables cifmw_openshift_login_kubeconfig={{ cifmw_openshift_login_kubeconfig | default(cifmw_openshift_kubeconfig) | default( ansible_env.KUBECONFIG if 'KUBECONFIG' in ansible_env else cifmw_openshift_login_kubeconfig_default_path ) | trim }}, cifmw_openshift_login_user={{ cifmw_openshift_login_user | default(cifmw_openshift_user) | default(omit) }}, cifmw_openshift_login_password={{********** cifmw_openshift_login_password | default(cifmw_openshift_password) | default(omit) }}, cifmw_openshift_login_api={{ cifmw_openshift_login_api | default(cifmw_openshift_api) | default(omit) }}, cifmw_openshift_login_cert_login={{ cifmw_openshift_login_cert_login | default(false)}}, cifmw_openshift_login_provided_token={{ cifmw_openshift_provided_token | default(omit) }}, cacheable=True] *** 2026-03-09 18:35:27,371 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.026) 0:02:23.307 ********** 2026-03-09 18:35:27,371 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.026) 0:02:23.305 ********** 2026-03-09 18:35:27,399 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:27,407 p=31718 u=zuul n=ansible | TASK [openshift_login : Check if kubeconfig exists path={{ cifmw_openshift_login_kubeconfig }}] *** 2026-03-09 18:35:27,407 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.035) 0:02:23.342 ********** 2026-03-09 18:35:27,407 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.035) 0:02:23.341 ********** 2026-03-09 18:35:27,596 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:27,617 p=31718 u=zuul n=ansible | TASK [openshift_login : Assert that enough data is provided to log in to OpenShift that=cifmw_openshift_login_kubeconfig_stat.stat.exists or (cifmw_openshift_login_provided_token is defined and cifmw_openshift_login_provided_token != '') or ( (cifmw_openshift_login_user is defined) and (cifmw_openshift_login_password is defined) and (cifmw_openshift_login_api is defined) ), msg=If an existing kubeconfig is not provided user/pwd or provided/initial token and API URL must be given] *** 2026-03-09 18:35:27,618 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.210) 0:02:23.553 ********** 2026-03-09 18:35:27,618 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.211) 0:02:23.552 ********** 2026-03-09 18:35:27,639 p=31718 u=zuul n=ansible | ok: [controller] => changed: false msg: All assertions passed 2026-03-09 18:35:27,646 p=31718 u=zuul n=ansible | TASK [openshift_login : Fetch kubeconfig content src={{ cifmw_openshift_login_kubeconfig }}] *** 2026-03-09 18:35:27,646 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.028) 0:02:23.582 ********** 2026-03-09 18:35:27,646 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.028) 0:02:23.580 ********** 2026-03-09 18:35:27,665 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:27,675 p=31718 u=zuul n=ansible | TASK [openshift_login : Fetch x509 key based users cifmw_openshift_login_key_based_users={{ ( cifmw_openshift_login_kubeconfig_content_b64.content | b64decode | from_yaml ). users | default([]) | selectattr('user.client-certificate-data', 'defined') | map(attribute="name") | map("split", "/") | map("first") }}, cacheable=True] *** 2026-03-09 18:35:27,675 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.029) 0:02:23.611 ********** 2026-03-09 18:35:27,675 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.029) 0:02:23.610 ********** 2026-03-09 18:35:27,691 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:27,698 p=31718 u=zuul n=ansible | TASK [openshift_login : Assign key based user if not provided and available cifmw_openshift_login_user={{ (cifmw_openshift_login_assume_cert_system_user | ternary('system:', '')) + (cifmw_openshift_login_key_based_users | map('replace', 'system:', '') | unique | first) }}, cifmw_openshift_login_cert_login=True, cacheable=True] *** 2026-03-09 18:35:27,698 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.023) 0:02:23.634 ********** 2026-03-09 18:35:27,698 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.023) 0:02:23.633 ********** 2026-03-09 18:35:27,712 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:27,719 p=31718 u=zuul n=ansible | TASK [openshift_login : Set the retry count cifmw_openshift_login_retries_cnt={{ 0 if cifmw_openshift_login_retries_cnt is undefined else cifmw_openshift_login_retries_cnt|int + 1 }}] *** 2026-03-09 18:35:27,719 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.021) 0:02:23.655 ********** 2026-03-09 18:35:27,719 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.021) 0:02:23.654 ********** 2026-03-09 18:35:27,743 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:27,751 p=31718 u=zuul n=ansible | TASK [openshift_login : Fetch token _raw_params=try_login.yml] ***************** 2026-03-09 18:35:27,751 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.031) 0:02:23.687 ********** 2026-03-09 18:35:27,751 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.031) 0:02:23.685 ********** 2026-03-09 18:35:27,769 p=31718 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/openshift_login/tasks/try_login.yml for controller 2026-03-09 18:35:27,780 p=31718 u=zuul n=ansible | TASK [openshift_login : Try get OpenShift access token _raw_params=oc whoami -t] *** 2026-03-09 18:35:27,780 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.028) 0:02:23.716 ********** 2026-03-09 18:35:27,780 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.028) 0:02:23.714 ********** 2026-03-09 18:35:27,794 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:27,801 p=31718 u=zuul n=ansible | TASK [openshift_login : Fetch OpenShift token output_dir={{ cifmw_openshift_login_basedir }}/artifacts, script=oc login {%- if cifmw_openshift_login_provided_token is not defined %} {%- if cifmw_openshift_login_user is defined %} -u {{ cifmw_openshift_login_user }} {%- endif %} {%- if cifmw_openshift_login_password is defined %} -p {{ cifmw_openshift_login_password }} {%- endif %} {% else %} --token={{ cifmw_openshift_login_provided_token }} {%- endif %} {%- if cifmw_openshift_login_skip_tls_verify|bool %} --insecure-skip-tls-verify=true {%- endif %} {%- if cifmw_openshift_login_api is defined %} {{ cifmw_openshift_login_api }} {%- endif %}] *** 2026-03-09 18:35:27,801 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.021) 0:02:23.737 ********** 2026-03-09 18:35:27,801 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:27 +0000 (0:00:00.021) 0:02:23.735 ********** 2026-03-09 18:35:27,894 p=31718 u=zuul n=ansible | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_002_fetch_openshift.log 2026-03-09 18:35:28,400 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:35:28,407 p=31718 u=zuul n=ansible | TASK [openshift_login : Ensure kubeconfig is provided that=cifmw_openshift_login_kubeconfig != ""] *** 2026-03-09 18:35:28,407 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:28 +0000 (0:00:00.606) 0:02:24.343 ********** 2026-03-09 18:35:28,407 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:28 +0000 (0:00:00.606) 0:02:24.342 ********** 2026-03-09 18:35:28,425 p=31718 u=zuul n=ansible | ok: [controller] => changed: false msg: All assertions passed 2026-03-09 18:35:28,433 p=31718 u=zuul n=ansible | TASK [openshift_login : Fetch new OpenShift access token _raw_params=oc whoami -t] *** 2026-03-09 18:35:28,433 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:28 +0000 (0:00:00.025) 0:02:24.369 ********** 2026-03-09 18:35:28,433 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:28 +0000 (0:00:00.025) 0:02:24.368 ********** 2026-03-09 18:35:28,727 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:35:28,735 p=31718 u=zuul n=ansible | TASK [openshift_login : Set new OpenShift token cifmw_openshift_login_token={{ (not cifmw_openshift_login_new_token_out.skipped | default(false)) | ternary(cifmw_openshift_login_new_token_out.stdout, cifmw_openshift_login_whoami_out.stdout) }}, cacheable=True] *** 2026-03-09 18:35:28,735 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:28 +0000 (0:00:00.301) 0:02:24.671 ********** 2026-03-09 18:35:28,735 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:28 +0000 (0:00:00.301) 0:02:24.669 ********** 2026-03-09 18:35:28,759 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:28,767 p=31718 u=zuul n=ansible | TASK [openshift_login : Fetch OpenShift API URL _raw_params=oc whoami --show-server=true] *** 2026-03-09 18:35:28,767 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:28 +0000 (0:00:00.032) 0:02:24.703 ********** 2026-03-09 18:35:28,767 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:28 +0000 (0:00:00.031) 0:02:24.701 ********** 2026-03-09 18:35:29,051 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:35:29,060 p=31718 u=zuul n=ansible | TASK [openshift_login : Fetch OpenShift kubeconfig context _raw_params=oc whoami -c] *** 2026-03-09 18:35:29,060 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:29 +0000 (0:00:00.292) 0:02:24.996 ********** 2026-03-09 18:35:29,060 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:29 +0000 (0:00:00.292) 0:02:24.994 ********** 2026-03-09 18:35:29,374 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:35:29,383 p=31718 u=zuul n=ansible | TASK [openshift_login : Fetch OpenShift current user _raw_params=oc whoami] **** 2026-03-09 18:35:29,384 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:29 +0000 (0:00:00.323) 0:02:25.319 ********** 2026-03-09 18:35:29,384 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:29 +0000 (0:00:00.323) 0:02:25.318 ********** 2026-03-09 18:35:29,726 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:35:29,736 p=31718 u=zuul n=ansible | TASK [openshift_login : Set OpenShift user, context and API facts cifmw_openshift_login_api={{ cifmw_openshift_login_api_out.stdout }}, cifmw_openshift_login_context={{ cifmw_openshift_login_context_out.stdout }}, cifmw_openshift_login_user={{ _oauth_user }}, cifmw_openshift_kubeconfig={{ cifmw_openshift_login_kubeconfig }}, cifmw_openshift_api={{ cifmw_openshift_login_api_out.stdout }}, cifmw_openshift_context={{ cifmw_openshift_login_context_out.stdout }}, cifmw_openshift_user={{ _oauth_user }}, cifmw_openshift_token={{ cifmw_openshift_login_token | default(omit) }}, cifmw_install_yamls_environment={{ ( cifmw_install_yamls_environment | combine({'KUBECONFIG': cifmw_openshift_login_kubeconfig}) ) if cifmw_install_yamls_environment is defined else omit }}, cacheable=True] *** 2026-03-09 18:35:29,736 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:29 +0000 (0:00:00.352) 0:02:25.672 ********** 2026-03-09 18:35:29,736 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:29 +0000 (0:00:00.352) 0:02:25.670 ********** 2026-03-09 18:35:29,762 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:29,770 p=31718 u=zuul n=ansible | TASK [openshift_login : Create the openshift_login parameters file dest={{ cifmw_basedir }}/artifacts/parameters/openshift-login-params.yml, content={{ cifmw_openshift_login_params_content | from_yaml | to_nice_yaml }}, mode=0600] *** 2026-03-09 18:35:29,770 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:29 +0000 (0:00:00.034) 0:02:25.706 ********** 2026-03-09 18:35:29,770 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:29 +0000 (0:00:00.034) 0:02:25.705 ********** 2026-03-09 18:35:30,288 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:35:30,297 p=31718 u=zuul n=ansible | TASK [openshift_login : Read the install yamls parameters file path={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml] *** 2026-03-09 18:35:30,297 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:30 +0000 (0:00:00.526) 0:02:26.233 ********** 2026-03-09 18:35:30,297 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:30 +0000 (0:00:00.526) 0:02:26.232 ********** 2026-03-09 18:35:30,636 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:30,645 p=31718 u=zuul n=ansible | TASK [openshift_login : Append the KUBECONFIG to the install yamls parameters content={{ cifmw_openshift_login_install_yamls_artifacts_slurp['content'] | b64decode | from_yaml | combine( { 'cifmw_install_yamls_environment': { 'KUBECONFIG': cifmw_openshift_login_kubeconfig } }, recursive=true) | to_nice_yaml }}, dest={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml, mode=0600] *** 2026-03-09 18:35:30,645 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:30 +0000 (0:00:00.347) 0:02:26.581 ********** 2026-03-09 18:35:30,645 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:30 +0000 (0:00:00.347) 0:02:26.579 ********** 2026-03-09 18:35:31,297 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:35:31,310 p=31718 u=zuul n=ansible | TASK [openshift_setup : Ensure output directory exists path={{ cifmw_openshift_setup_basedir }}/artifacts, state=directory, mode=0755] *** 2026-03-09 18:35:31,311 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:31 +0000 (0:00:00.665) 0:02:27.246 ********** 2026-03-09 18:35:31,311 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:31 +0000 (0:00:00.665) 0:02:27.245 ********** 2026-03-09 18:35:31,545 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:31,555 p=31718 u=zuul n=ansible | TASK [openshift_setup : Fetch namespaces to create cifmw_openshift_setup_namespaces={{ (( ([cifmw_install_yamls_defaults['NAMESPACE']] + ([cifmw_install_yamls_defaults['OPERATOR_NAMESPACE']] if 'OPERATOR_NAMESPACE' is in cifmw_install_yamls_defaults else []) ) if cifmw_install_yamls_defaults is defined else [] ) + cifmw_openshift_setup_create_namespaces) | unique }}] *** 2026-03-09 18:35:31,555 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:31 +0000 (0:00:00.244) 0:02:27.491 ********** 2026-03-09 18:35:31,555 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:31 +0000 (0:00:00.244) 0:02:27.490 ********** 2026-03-09 18:35:31,576 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:31,592 p=31718 u=zuul n=ansible | TASK [openshift_setup : Create required namespaces kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit) }}, name={{ item }}, kind=Namespace, state=present] *** 2026-03-09 18:35:31,592 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:31 +0000 (0:00:00.036) 0:02:27.527 ********** 2026-03-09 18:35:31,592 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:31 +0000 (0:00:00.036) 0:02:27.526 ********** 2026-03-09 18:35:33,029 p=31718 u=zuul n=ansible | changed: [controller] => (item=openstack) 2026-03-09 18:35:33,823 p=31718 u=zuul n=ansible | changed: [controller] => (item=openstack-operators) 2026-03-09 18:35:33,835 p=31718 u=zuul n=ansible | TASK [openshift_setup : Get internal OpenShift registry route kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, kind=Route, name=default-route, namespace=openshift-image-registry] *** 2026-03-09 18:35:33,835 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:02.243) 0:02:29.771 ********** 2026-03-09 18:35:33,835 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:02.243) 0:02:29.769 ********** 2026-03-09 18:35:33,846 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:33,854 p=31718 u=zuul n=ansible | TASK [openshift_setup : Allow anonymous image-pulls in CRC registry for targeted namespaces state=present, kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'kind': 'RoleBinding', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'metadata': {'name': 'system:image-puller', 'namespace': '{{ item }}'}, 'subjects': [{'kind': 'User', 'name': 'system:anonymous'}, {'kind': 'User', 'name': 'system:unauthenticated'}], 'roleRef': {'kind': 'ClusterRole', 'name': 'system:image-puller'}}] *** 2026-03-09 18:35:33,854 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:00.019) 0:02:29.790 ********** 2026-03-09 18:35:33,854 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:00.019) 0:02:29.789 ********** 2026-03-09 18:35:33,872 p=31718 u=zuul n=ansible | skipping: [controller] => (item=openstack) 2026-03-09 18:35:33,872 p=31718 u=zuul n=ansible | skipping: [controller] => (item=openstack-operators) 2026-03-09 18:35:33,873 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:33,880 p=31718 u=zuul n=ansible | TASK [openshift_setup : Wait for the image registry to be ready kind=Deployment, name=image-registry, namespace=openshift-image-registry, kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, wait=True, wait_sleep=10, wait_timeout=600, wait_condition={'type': 'Available', 'status': 'True'}] *** 2026-03-09 18:35:33,881 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:00.026) 0:02:29.816 ********** 2026-03-09 18:35:33,881 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:00.026) 0:02:29.815 ********** 2026-03-09 18:35:33,900 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:33,908 p=31718 u=zuul n=ansible | TASK [openshift_setup : Login into OpenShift internal registry output_dir={{ cifmw_openshift_setup_basedir }}/artifacts, script=podman login -u {{ cifmw_openshift_user }} -p {{ cifmw_openshift_token }} {%- if cifmw_openshift_setup_skip_internal_registry_tls_verify|bool %} --tls-verify=false {%- endif %} {{ cifmw_openshift_setup_registry_default_route.resources[0].spec.host }}] *** 2026-03-09 18:35:33,908 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:00.027) 0:02:29.844 ********** 2026-03-09 18:35:33,908 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:00.027) 0:02:29.843 ********** 2026-03-09 18:35:33,928 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:33,936 p=31718 u=zuul n=ansible | TASK [Ensure we have custom CA installed on host role=install_ca] ************** 2026-03-09 18:35:33,936 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:00.028) 0:02:29.872 ********** 2026-03-09 18:35:33,937 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:00.028) 0:02:29.871 ********** 2026-03-09 18:35:33,954 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:33,962 p=31718 u=zuul n=ansible | TASK [openshift_setup : Update ca bundle _raw_params=update-ca-trust extract] *** 2026-03-09 18:35:33,962 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:00.025) 0:02:29.897 ********** 2026-03-09 18:35:33,962 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:00.025) 0:02:29.896 ********** 2026-03-09 18:35:33,979 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:33,988 p=31718 u=zuul n=ansible | TASK [openshift_setup : Slurp CAs file src={{ cifmw_openshift_setup_ca_bundle_path }}] *** 2026-03-09 18:35:33,988 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:00.026) 0:02:29.924 ********** 2026-03-09 18:35:33,988 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:33 +0000 (0:00:00.026) 0:02:29.922 ********** 2026-03-09 18:35:34,005 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:34,012 p=31718 u=zuul n=ansible | TASK [openshift_setup : Create config map with registry CAs kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'apiVersion': 'v1', 'kind': 'ConfigMap', 'metadata': {'namespace': 'openshift-config', 'name': 'registry-cas'}, 'data': '{{ _config_map_data | items2dict }}'}] *** 2026-03-09 18:35:34,012 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:34 +0000 (0:00:00.024) 0:02:29.948 ********** 2026-03-09 18:35:34,012 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:34 +0000 (0:00:00.024) 0:02:29.946 ********** 2026-03-09 18:35:34,029 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:34,037 p=31718 u=zuul n=ansible | TASK [openshift_setup : Install Red Hat CA for pulling images from internal registry kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, merge_type=merge, definition={'apiVersion': 'config.openshift.io/v1', 'kind': 'Image', 'metadata': {'name': 'cluster'}, 'spec': {'additionalTrustedCA': {'name': 'registry-cas'}}}] *** 2026-03-09 18:35:34,038 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:34 +0000 (0:00:00.025) 0:02:29.973 ********** 2026-03-09 18:35:34,038 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:34 +0000 (0:00:00.025) 0:02:29.972 ********** 2026-03-09 18:35:34,055 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:34,065 p=31718 u=zuul n=ansible | TASK [openshift_setup : Add insecure registry kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, merge_type=merge, definition={'apiVersion': 'config.openshift.io/v1', 'kind': 'Image', 'metadata': {'name': 'cluster'}, 'spec': {'registrySources': {'insecureRegistries': ['{{ cifmw_update_containers_registry }}'], 'allowedRegistries': '{{ all_registries }}'}}}] *** 2026-03-09 18:35:34,065 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:34 +0000 (0:00:00.027) 0:02:30.001 ********** 2026-03-09 18:35:34,065 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:34 +0000 (0:00:00.027) 0:02:29.999 ********** 2026-03-09 18:35:34,082 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:34,091 p=31718 u=zuul n=ansible | TASK [openshift_setup : Create a ICSP with repository digest mirrors kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'apiVersion': 'operator.openshift.io/v1alpha1', 'kind': 'ImageContentSourcePolicy', 'metadata': {'name': 'registry-digest-mirrors'}, 'spec': {'repositoryDigestMirrors': '{{ cifmw_openshift_setup_digest_mirrors }}'}}] *** 2026-03-09 18:35:34,091 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:34 +0000 (0:00:00.025) 0:02:30.027 ********** 2026-03-09 18:35:34,091 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:34 +0000 (0:00:00.025) 0:02:30.025 ********** 2026-03-09 18:35:34,113 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:34,122 p=31718 u=zuul n=ansible | TASK [openshift_setup : Gather network.operator info kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, api_version=operator.openshift.io/v1, kind=Network, name=cluster] *** 2026-03-09 18:35:34,122 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:34 +0000 (0:00:00.030) 0:02:30.058 ********** 2026-03-09 18:35:34,122 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:34 +0000 (0:00:00.030) 0:02:30.056 ********** 2026-03-09 18:35:35,165 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:35:35,179 p=31718 u=zuul n=ansible | TASK [openshift_setup : Patch network operator api_version=operator.openshift.io/v1, kubeconfig={{ cifmw_openshift_kubeconfig }}, kind=Network, name=cluster, persist_config=True, patch=[{'path': '/spec/defaultNetwork/ovnKubernetesConfig/gatewayConfig/routingViaHost', 'value': True, 'op': 'replace'}, {'path': '/spec/defaultNetwork/ovnKubernetesConfig/gatewayConfig/ipForwarding', 'value': 'Global', 'op': 'replace'}]] *** 2026-03-09 18:35:35,180 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:35 +0000 (0:00:01.057) 0:02:31.115 ********** 2026-03-09 18:35:35,180 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:35 +0000 (0:00:01.057) 0:02:31.114 ********** 2026-03-09 18:35:36,252 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:35:36,263 p=31718 u=zuul n=ansible | TASK [openshift_setup : Patch samples registry configuration kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, api_version=samples.operator.openshift.io/v1, kind=Config, name=cluster, patch=[{'op': 'replace', 'path': '/spec/samplesRegistry', 'value': '{{ cifmw_openshift_setup_samples_registry }}'}]] *** 2026-03-09 18:35:36,263 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:36 +0000 (0:00:01.083) 0:02:32.199 ********** 2026-03-09 18:35:36,263 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:36 +0000 (0:00:01.083) 0:02:32.198 ********** 2026-03-09 18:35:37,143 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:35:37,150 p=31718 u=zuul n=ansible | TASK [openshift_setup : Delete the pods from openshift-marketplace namespace kind=Pod, state=absent, delete_all=True, kubeconfig={{ cifmw_openshift_kubeconfig }}, namespace=openshift-marketplace] *** 2026-03-09 18:35:37,150 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:37 +0000 (0:00:00.887) 0:02:33.086 ********** 2026-03-09 18:35:37,151 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:37 +0000 (0:00:00.887) 0:02:33.085 ********** 2026-03-09 18:35:37,168 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:37,176 p=31718 u=zuul n=ansible | TASK [openshift_setup : Wait for openshift-marketplace pods to be running _raw_params=oc wait pod --all --for=condition=Ready -n openshift-marketplace --timeout=1m] *** 2026-03-09 18:35:37,176 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:37 +0000 (0:00:00.025) 0:02:33.112 ********** 2026-03-09 18:35:37,176 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:37 +0000 (0:00:00.025) 0:02:33.110 ********** 2026-03-09 18:35:37,192 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:35:37,209 p=31718 u=zuul n=ansible | TASK [Deploy Observability operator. name=openshift_obs] *********************** 2026-03-09 18:35:37,209 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:37 +0000 (0:00:00.033) 0:02:33.145 ********** 2026-03-09 18:35:37,209 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:37 +0000 (0:00:00.033) 0:02:33.144 ********** 2026-03-09 18:35:37,288 p=31718 u=zuul n=ansible | TASK [openshift_obs : Install cluster observability operator. definition={{cifmw_openshift_obs_definition }}, kubeconfig={{ cifmw_openshift_kubeconfig }}, state=present] *** 2026-03-09 18:35:37,289 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:37 +0000 (0:00:00.079) 0:02:33.224 ********** 2026-03-09 18:35:37,289 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:37 +0000 (0:00:00.079) 0:02:33.223 ********** 2026-03-09 18:35:38,125 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:35:38,133 p=31718 u=zuul n=ansible | TASK [openshift_obs : Wait for observability operator deployment kind=Deployment, namespace=openshift-operators, name=observability-operator, wait=True, wait_timeout=300, wait_condition={'type': 'Available', 'status': 'True'}, kubeconfig={{ cifmw_openshift_kubeconfig }}] *** 2026-03-09 18:35:38,133 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:38 +0000 (0:00:00.844) 0:02:34.068 ********** 2026-03-09 18:35:38,133 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:35:38 +0000 (0:00:00.844) 0:02:34.067 ********** 2026-03-09 18:36:24,465 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:36:24,477 p=31718 u=zuul n=ansible | TASK [openshift_obs : Wait for observability-operator pod kind=Pod, namespace=openshift-operators, label_selectors=['app.kubernetes.io/name = observability-operator'], wait=True, wait_timeout=300, wait_condition={'type': 'Ready', 'status': 'True'}, kubeconfig={{ cifmw_openshift_kubeconfig }}] *** 2026-03-09 18:36:24,477 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:24 +0000 (0:00:46.344) 0:03:20.413 ********** 2026-03-09 18:36:24,477 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:24 +0000 (0:00:46.344) 0:03:20.411 ********** 2026-03-09 18:36:25,435 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:36:25,453 p=31718 u=zuul n=ansible | TASK [Deploy Metal3 BMHs name=deploy_bmh] ************************************** 2026-03-09 18:36:25,454 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.976) 0:03:21.389 ********** 2026-03-09 18:36:25,454 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.976) 0:03:21.388 ********** 2026-03-09 18:36:25,473 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:36:25,481 p=31718 u=zuul n=ansible | TASK [Install certmanager operator role name=cert_manager] ********************* 2026-03-09 18:36:25,482 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.027) 0:03:21.417 ********** 2026-03-09 18:36:25,482 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.027) 0:03:21.416 ********** 2026-03-09 18:36:25,502 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:36:25,512 p=31718 u=zuul n=ansible | TASK [Configure hosts networking using nmstate name=ci_nmstate] **************** 2026-03-09 18:36:25,512 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.030) 0:03:21.447 ********** 2026-03-09 18:36:25,512 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.030) 0:03:21.446 ********** 2026-03-09 18:36:25,530 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:36:25,537 p=31718 u=zuul n=ansible | TASK [Configure multus networks name=ci_multus] ******************************** 2026-03-09 18:36:25,537 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.025) 0:03:21.473 ********** 2026-03-09 18:36:25,537 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.025) 0:03:21.471 ********** 2026-03-09 18:36:25,549 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:36:25,557 p=31718 u=zuul n=ansible | TASK [Deploy Sushy Emulator service pod name=sushy_emulator] ******************* 2026-03-09 18:36:25,557 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.020) 0:03:21.493 ********** 2026-03-09 18:36:25,557 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.020) 0:03:21.492 ********** 2026-03-09 18:36:25,573 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:36:25,580 p=31718 u=zuul n=ansible | TASK [Setup Libvirt on controller name=libvirt_manager] ************************ 2026-03-09 18:36:25,580 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.022) 0:03:21.516 ********** 2026-03-09 18:36:25,580 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.022) 0:03:21.514 ********** 2026-03-09 18:36:25,595 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:36:25,603 p=31718 u=zuul n=ansible | TASK [Prepare container package builder name=pkg_build] ************************ 2026-03-09 18:36:25,603 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.023) 0:03:21.539 ********** 2026-03-09 18:36:25,603 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.023) 0:03:21.537 ********** 2026-03-09 18:36:25,621 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:36:25,629 p=31718 u=zuul n=ansible | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] *** 2026-03-09 18:36:25,629 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.025) 0:03:21.565 ********** 2026-03-09 18:36:25,629 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.025) 0:03:21.563 ********** 2026-03-09 18:36:25,678 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:36:25,686 p=31718 u=zuul n=ansible | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] *** 2026-03-09 18:36:25,687 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.057) 0:03:21.622 ********** 2026-03-09 18:36:25,687 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.057) 0:03:21.621 ********** 2026-03-09 18:36:25,782 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:36:25,794 p=31718 u=zuul n=ansible | TASK [run_hook : Loop on hooks for post_infra _raw_params={{ hook.type }}.yml] *** 2026-03-09 18:36:25,794 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.107) 0:03:21.730 ********** 2026-03-09 18:36:25,794 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.107) 0:03:21.728 ********** 2026-03-09 18:36:25,891 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:36:25,909 p=31718 u=zuul n=ansible | TASK [Load parameters dir={{ item }}] ****************************************** 2026-03-09 18:36:25,909 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.115) 0:03:21.845 ********** 2026-03-09 18:36:25,909 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:25 +0000 (0:00:00.115) 0:03:21.844 ********** 2026-03-09 18:36:25,997 p=31718 u=zuul n=ansible | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters) 2026-03-09 18:36:26,001 p=31718 u=zuul n=ansible | ok: [controller] => (item=/etc/ci/env) 2026-03-09 18:36:26,053 p=31718 u=zuul n=ansible | TASK [Ensure that the isolated net was configured for crc that=['crc_ci_bootstrap_networks_out is defined', 'crc_ci_bootstrap_networks_out[_crc_hostname] is defined', "crc_ci_bootstrap_networks_out[_crc_hostname]['default'] is defined"]] *** 2026-03-09 18:36:26,053 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.143) 0:03:21.989 ********** 2026-03-09 18:36:26,053 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.143) 0:03:21.987 ********** 2026-03-09 18:36:26,081 p=31718 u=zuul n=ansible | ok: [controller] => changed: false msg: All assertions passed 2026-03-09 18:36:26,088 p=31718 u=zuul n=ansible | TASK [Set facts for further usage within the framework cifmw_edpm_prepare_extra_vars={'NNCP_INTERFACE': '{{ crc_ci_bootstrap_networks_out.crc.default.iface }}', 'NETWORK_MTU': '{{ crc_ci_bootstrap_networks_out.crc.default.mtu }}', 'NNCP_DNS_SERVER': "{{\n cifmw_nncp_dns_server |\n default(crc_ci_bootstrap_networks_out[_crc_hostname].default.ip) |\n split('/') | first\n}}"}] *** 2026-03-09 18:36:26,088 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.035) 0:03:22.024 ********** 2026-03-09 18:36:26,088 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.035) 0:03:22.023 ********** 2026-03-09 18:36:26,106 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:36:26,131 p=31718 u=zuul n=ansible | PLAY [Deploy Openstack Operators] ********************************************** 2026-03-09 18:36:26,147 p=31718 u=zuul n=ansible | TASK [Use the locally built operators if any _local_operators_indexes={{ _local_operators_indexes|default({}) | combine({ item.key.split('-')[0]|upper+'_IMG': cifmw_operator_build_output['operators'][item.key].image_catalog}) }}] *** 2026-03-09 18:36:26,148 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.059) 0:03:22.083 ********** 2026-03-09 18:36:26,148 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.059) 0:03:22.082 ********** 2026-03-09 18:36:26,181 p=31718 u=zuul n=ansible | ok: [controller] => (item={'key': 'openstack-operator', 'value': {'git_commit_hash': '581e9c0db82d502a54d54052822ffac7bccc0534', 'git_src_dir': '/home/zuul-worker/src/github.com/openstack-k8s-operators/openstack-operator', 'image': '38.102.83.110:5001/openstack-k8s-operators/openstack-operator:581e9c0db82d502a54d54052822ffac7bccc0534', 'image_bundle': '38.102.83.110:5001/openstack-k8s-operators/openstack-operator-bundle:581e9c0db82d502a54d54052822ffac7bccc0534', 'image_catalog': '38.102.83.110:5001/openstack-k8s-operators/openstack-operator-index:581e9c0db82d502a54d54052822ffac7bccc0534'}}) 2026-03-09 18:36:26,192 p=31718 u=zuul n=ansible | ok: [controller] => (item={'key': 'watcher-operator', 'value': {'git_commit_hash': '2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'git_src_dir': '/home/zuul-worker/src/github.com/openstack-k8s-operators/watcher-operator', 'image': '38.102.83.110:5001/openstack-k8s-operators/watcher-operator:2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'image_bundle': '38.102.83.110:5001/openstack-k8s-operators/watcher-operator-bundle:2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'image_catalog': '38.102.83.110:5001/openstack-k8s-operators/watcher-operator-index:2e035aad6e396aeb72cc6aec8684c43e59f8b674'}}) 2026-03-09 18:36:26,199 p=31718 u=zuul n=ansible | TASK [Set install_yamls Makefile environment variables cifmw_edpm_prepare_common_env={{ cifmw_install_yamls_environment | combine({'PATH': cifmw_path}) | combine(cifmw_edpm_prepare_extra_vars | default({})) }}, cifmw_edpm_prepare_operators_build_output={{ operators_build_output }}, cifmw_edpm_prepare_make_openstack_env={{ _local_operators_indexes | combine(_openstack_operator_images) }}] *** 2026-03-09 18:36:26,199 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.051) 0:03:22.135 ********** 2026-03-09 18:36:26,199 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.051) 0:03:22.134 ********** 2026-03-09 18:36:26,219 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:36:26,226 p=31718 u=zuul n=ansible | TASK [detect if openstack operator is installed _raw_params=oc get sub --ignore-not-found=true -n openstack-operators -o name openstack-operator] *** 2026-03-09 18:36:26,226 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.027) 0:03:22.162 ********** 2026-03-09 18:36:26,227 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.027) 0:03:22.161 ********** 2026-03-09 18:36:26,568 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:36:26,575 p=31718 u=zuul n=ansible | TASK [Install openstack operator and wait for the csv to succeed name=install_yamls_makes, tasks_from=make_openstack_init] *** 2026-03-09 18:36:26,575 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.348) 0:03:22.511 ********** 2026-03-09 18:36:26,575 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.348) 0:03:22.510 ********** 2026-03-09 18:36:26,603 p=31718 u=zuul n=ansible | TASK [install_yamls_makes : Debug make_openstack_init_env var=make_openstack_init_env] *** 2026-03-09 18:36:26,604 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.028) 0:03:22.539 ********** 2026-03-09 18:36:26,604 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.028) 0:03:22.538 ********** 2026-03-09 18:36:26,631 p=31718 u=zuul n=ansible | ok: [controller] => make_openstack_init_env: BMO_SETUP: false CHECKOUT_FROM_OPENSTACK_REF: 'true' KUBECONFIG: /home/zuul/.crc/machines/crc/kubeconfig NETWORK_MTU: 1500 NNCP_DNS_SERVER: 192.168.122.10 NNCP_INTERFACE: ens7 OPENSTACK_BUNDLE_IMG: 38.102.83.110:5001/openstack-k8s-operators/openstack-operator-bundle:581e9c0db82d502a54d54052822ffac7bccc0534 OPENSTACK_IMG: 38.102.83.110:5001/openstack-k8s-operators/openstack-operator-index:581e9c0db82d502a54d54052822ffac7bccc0534 OPENSTACK_K8S_BRANCH: main OUT: /home/zuul/ci-framework-data/artifacts/manifests OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm PATH: /home/zuul/.crc/bin:/home/zuul/.crc/bin/oc:/home/zuul/bin:/home/zuul/.local/bin:/home/zuul/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin WATCHER_BRANCH: '' WATCHER_IMG: 38.102.83.110:5001/openstack-k8s-operators/watcher-operator-index:2e035aad6e396aeb72cc6aec8684c43e59f8b674 WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator 2026-03-09 18:36:26,643 p=31718 u=zuul n=ansible | TASK [install_yamls_makes : Debug make_openstack_init_params var=make_openstack_init_params] *** 2026-03-09 18:36:26,643 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.039) 0:03:22.579 ********** 2026-03-09 18:36:26,643 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.039) 0:03:22.577 ********** 2026-03-09 18:36:26,660 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:36:26,667 p=31718 u=zuul n=ansible | TASK [install_yamls_makes : Run openstack_init output_dir={{ cifmw_basedir }}/artifacts, chdir=/home/zuul/src/github.com/openstack-k8s-operators/install_yamls, script=make openstack_init, dry_run={{ make_openstack_init_dryrun|default(false)|bool }}, extra_args={{ dict((make_openstack_init_env|default({})), **(make_openstack_init_params|default({}))) }}] *** 2026-03-09 18:36:26,667 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.024) 0:03:22.603 ********** 2026-03-09 18:36:26,667 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:36:26 +0000 (0:00:00.024) 0:03:22.602 ********** 2026-03-09 18:36:26,775 p=31718 u=zuul n=ansible | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_003_run_openstack.log 2026-03-09 18:41:14,098 p=31718 u=zuul n=ansible | [WARNING]: conditional statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ make_openstack_init_until | default(true) }} 2026-03-09 18:41:14,101 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:41:14,118 p=31718 u=zuul n=ansible | TASK [Run hooks after installing openstack name=run_hook] ********************** 2026-03-09 18:41:14,118 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:04:47.450) 0:08:10.053 ********** 2026-03-09 18:41:14,118 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:04:47.450) 0:08:10.052 ********** 2026-03-09 18:41:14,155 p=31718 u=zuul n=ansible | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] *** 2026-03-09 18:41:14,155 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.037) 0:08:10.091 ********** 2026-03-09 18:41:14,155 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.037) 0:08:10.089 ********** 2026-03-09 18:41:14,227 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:41:14,238 p=31718 u=zuul n=ansible | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] *** 2026-03-09 18:41:14,238 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.083) 0:08:10.174 ********** 2026-03-09 18:41:14,238 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.083) 0:08:10.173 ********** 2026-03-09 18:41:14,327 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:41:14,334 p=31718 u=zuul n=ansible | TASK [run_hook : Loop on hooks for post_install_operators_kuttl_from_operator _raw_params={{ hook.type }}.yml] *** 2026-03-09 18:41:14,334 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.096) 0:08:10.270 ********** 2026-03-09 18:41:14,334 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.095) 0:08:10.269 ********** 2026-03-09 18:41:14,453 p=31718 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/run_hook/tasks/playbook.yml for controller => (item={'name': 'Deploy watcher operator', 'type': 'playbook', 'source': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator/ci/playbooks/deploy_watcher_operator.yaml', 'extra_vars': {'content_provider_os_registry_url': '38.102.83.110:5001/podified-master-centos10', 'watcher_catalog_image': '38.102.83.110:5001/openstack-k8s-operators/watcher-operator-index:2e035aad6e396aeb72cc6aec8684c43e59f8b674', 'watcher_services_tag': 'watcher_latest', 'watcher_repo': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator'}}) 2026-03-09 18:41:14,468 p=31718 u=zuul n=ansible | TASK [run_hook : Set playbook path for Deploy watcher operator cifmw_basedir={{ _bdir }}, hook_name={{ _hook_name }}, playbook_path={{ _play | realpath }}, log_path={{ _bdir }}/logs/{{ step }}_{{ _hook_name }}.log, extra_vars=-e namespace={{ cifmw_openstack_namespace }} {%- if hook.extra_vars is defined and hook.extra_vars|length > 0 -%} {% for key,value in hook.extra_vars.items() -%} {%- if key == 'file' %} -e "@{{ value }}" {%- else %} -e "{{ key }}={{ value }}" {%- endif %} {%- endfor %} {%- endif %}] *** 2026-03-09 18:41:14,468 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.134) 0:08:10.404 ********** 2026-03-09 18:41:14,469 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.134) 0:08:10.403 ********** 2026-03-09 18:41:14,515 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:41:14,524 p=31718 u=zuul n=ansible | TASK [run_hook : Get file stat path={{ playbook_path }}] *********************** 2026-03-09 18:41:14,525 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.056) 0:08:10.460 ********** 2026-03-09 18:41:14,525 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.056) 0:08:10.459 ********** 2026-03-09 18:41:14,742 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:41:14,748 p=31718 u=zuul n=ansible | TASK [run_hook : Fail if playbook doesn't exist msg=Playbook {{ playbook_path }} doesn't seem to exist.] *** 2026-03-09 18:41:14,749 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.223) 0:08:10.684 ********** 2026-03-09 18:41:14,749 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.223) 0:08:10.683 ********** 2026-03-09 18:41:14,764 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:41:14,770 p=31718 u=zuul n=ansible | TASK [run_hook : Get parameters files paths={{ (cifmw_basedir, 'artifacts/parameters') | path_join }}, file_type=file, patterns=*.yml] *** 2026-03-09 18:41:14,771 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.021) 0:08:10.706 ********** 2026-03-09 18:41:14,771 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.021) 0:08:10.705 ********** 2026-03-09 18:41:14,972 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:41:14,980 p=31718 u=zuul n=ansible | TASK [run_hook : Add parameters artifacts as extra variables extra_vars={{ extra_vars }} {% for file in cifmw_run_hook_parameters_files.files %} -e "@{{ file.path }}" {%- endfor %}] *** 2026-03-09 18:41:14,980 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.209) 0:08:10.916 ********** 2026-03-09 18:41:14,980 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:14 +0000 (0:00:00.209) 0:08:10.914 ********** 2026-03-09 18:41:14,997 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:41:15,004 p=31718 u=zuul n=ansible | TASK [run_hook : Ensure log directory exists path={{ log_path | dirname }}, state=directory, mode=0755] *** 2026-03-09 18:41:15,004 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:15 +0000 (0:00:00.024) 0:08:10.940 ********** 2026-03-09 18:41:15,004 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:15 +0000 (0:00:00.024) 0:08:10.939 ********** 2026-03-09 18:41:15,209 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:41:15,218 p=31718 u=zuul n=ansible | TASK [run_hook : Ensure artifacts directory exists path={{ cifmw_basedir }}/artifacts, state=directory, mode=0755] *** 2026-03-09 18:41:15,218 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:15 +0000 (0:00:00.214) 0:08:11.154 ********** 2026-03-09 18:41:15,219 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:15 +0000 (0:00:00.214) 0:08:11.153 ********** 2026-03-09 18:41:15,402 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:41:15,416 p=31718 u=zuul n=ansible | TASK [run_hook : Run hook without retry - Deploy watcher operator] ************* 2026-03-09 18:41:15,417 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:15 +0000 (0:00:00.198) 0:08:11.352 ********** 2026-03-09 18:41:15,417 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:41:15 +0000 (0:00:00.198) 0:08:11.351 ********** 2026-03-09 18:41:15,523 p=31718 u=zuul n=ansible | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_004_run_hook_without_retry_deploy.log 2026-03-09 18:42:15,252 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:42:15,262 p=31718 u=zuul n=ansible | TASK [run_hook : Run hook with retry - Deploy watcher operator] **************** 2026-03-09 18:42:15,262 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:42:15 +0000 (0:00:59.845) 0:09:11.198 ********** 2026-03-09 18:42:15,262 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:42:15 +0000 (0:00:59.845) 0:09:11.196 ********** 2026-03-09 18:42:15,278 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:42:15,285 p=31718 u=zuul n=ansible | TASK [run_hook : Check if we have a file path={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] *** 2026-03-09 18:42:15,285 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:42:15 +0000 (0:00:00.023) 0:09:11.221 ********** 2026-03-09 18:42:15,286 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:42:15 +0000 (0:00:00.023) 0:09:11.220 ********** 2026-03-09 18:42:15,500 p=31718 u=zuul n=ansible | ok: [controller] 2026-03-09 18:42:15,508 p=31718 u=zuul n=ansible | TASK [run_hook : Load generated content in main playbook file={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] *** 2026-03-09 18:42:15,508 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:42:15 +0000 (0:00:00.222) 0:09:11.444 ********** 2026-03-09 18:42:15,508 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:42:15 +0000 (0:00:00.222) 0:09:11.443 ********** 2026-03-09 18:42:15,521 p=31718 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:42:15,535 p=31718 u=zuul n=ansible | TASK [install kuttl test_suite dependencies chdir={{ ansible_user_dir }}/{{ operator_basedir }}, _raw_params=make kuttl-test-prep] *** 2026-03-09 18:42:15,535 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:42:15 +0000 (0:00:00.026) 0:09:11.471 ********** 2026-03-09 18:42:15,535 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:42:15 +0000 (0:00:00.026) 0:09:11.469 ********** 2026-03-09 18:44:55,386 p=31718 u=zuul n=ansible | changed: [controller] 2026-03-09 18:44:55,410 p=31718 u=zuul n=ansible | PLAY RECAP ********************************************************************* 2026-03-09 18:44:55,410 p=31718 u=zuul n=ansible | controller : ok=116 changed=42 unreachable=0 failed=0 skipped=77 rescued=0 ignored=0 2026-03-09 18:44:55,410 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:44:55 +0000 (0:02:39.875) 0:11:51.346 ********** 2026-03-09 18:44:55,410 p=31718 u=zuul n=ansible | =============================================================================== 2026-03-09 18:44:55,410 p=31718 u=zuul n=ansible | install_yamls_makes : Run openstack_init ------------------------------ 287.45s 2026-03-09 18:44:55,410 p=31718 u=zuul n=ansible | install kuttl test_suite dependencies --------------------------------- 159.88s 2026-03-09 18:44:55,410 p=31718 u=zuul n=ansible | run_hook : Run hook without retry - Deploy watcher operator ------------ 59.85s 2026-03-09 18:44:55,410 p=31718 u=zuul n=ansible | openshift_obs : Wait for observability operator deployment ------------- 46.34s 2026-03-09 18:44:55,410 p=31718 u=zuul n=ansible | install_yamls_makes : Run download_tools ------------------------------- 40.94s 2026-03-09 18:44:55,410 p=31718 u=zuul n=ansible | run_hook : Run hook without retry - Download needed tools -------------- 35.78s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | ci_setup : Install needed packages ------------------------------------- 30.69s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | repo_setup : Initialize python venv and install requirements ------------ 7.93s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | ci_setup : Install openshift client ------------------------------------- 5.70s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | openshift_setup : Create required namespaces ---------------------------- 2.24s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | ci_setup : Manage directories ------------------------------------------- 1.17s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | Gathering Facts --------------------------------------------------------- 1.14s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | repo_setup : Ensure directories are present ----------------------------- 1.10s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | openshift_setup : Patch network operator -------------------------------- 1.08s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | openshift_setup : Gather network.operator info -------------------------- 1.06s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | repo_setup : Make sure git-core package is installed -------------------- 1.04s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | openshift_obs : Wait for observability-operator pod --------------------- 0.98s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | install_yamls : Create the install_yamls parameters file ---------------- 0.94s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | openshift_setup : Patch samples registry configuration ------------------ 0.89s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | repo_setup : Get repo-setup repository ---------------------------------- 0.87s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | Monday 09 March 2026 18:44:55 +0000 (0:02:39.875) 0:11:51.345 ********** 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | =============================================================================== 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | install_yamls_makes --------------------------------------------------- 328.50s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | ansible.builtin.command ----------------------------------------------- 160.22s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | run_hook --------------------------------------------------------------- 98.98s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | openshift_obs ---------------------------------------------------------- 48.17s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | ci_setup --------------------------------------------------------------- 38.97s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | repo_setup ------------------------------------------------------------- 18.10s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | openshift_setup --------------------------------------------------------- 5.90s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | openshift_login --------------------------------------------------------- 4.26s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | install_yamls ----------------------------------------------------------- 3.93s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | cifmw_setup ------------------------------------------------------------- 1.52s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | gather_facts ------------------------------------------------------------ 1.14s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | install_ca -------------------------------------------------------------- 0.49s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | discover_latest_image --------------------------------------------------- 0.40s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | networking_mapper ------------------------------------------------------- 0.28s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | ansible.builtin.include_vars -------------------------------------------- 0.14s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | ansible.builtin.set_fact ------------------------------------------------ 0.14s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | ansible.builtin.include_role -------------------------------------------- 0.10s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | ansible.builtin.assert -------------------------------------------------- 0.07s 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2026-03-09 18:44:55,411 p=31718 u=zuul n=ansible | total ----------------------------------------------------------------- 711.31s 2026-03-09 18:44:59,213 p=40838 u=zuul n=ansible | PLAY [controller] ************************************************************** 2026-03-09 18:44:59,242 p=40838 u=zuul n=ansible | TASK [Run hooks before running kuttl tests name=run_hook] ********************** 2026-03-09 18:44:59,242 p=40838 u=zuul n=ansible | Monday 09 March 2026 18:44:59 +0000 (0:00:00.044) 0:00:00.045 ********** 2026-03-09 18:44:59,242 p=40838 u=zuul n=ansible | Monday 09 March 2026 18:44:59 +0000 (0:00:00.044) 0:00:00.044 ********** 2026-03-09 18:44:59,277 p=40838 u=zuul n=ansible | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] *** 2026-03-09 18:44:59,277 p=40838 u=zuul n=ansible | Monday 09 March 2026 18:44:59 +0000 (0:00:00.034) 0:00:00.079 ********** 2026-03-09 18:44:59,277 p=40838 u=zuul n=ansible | Monday 09 March 2026 18:44:59 +0000 (0:00:00.034) 0:00:00.079 ********** 2026-03-09 18:44:59,340 p=40838 u=zuul n=ansible | ok: [controller] 2026-03-09 18:44:59,352 p=40838 u=zuul n=ansible | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] *** 2026-03-09 18:44:59,352 p=40838 u=zuul n=ansible | Monday 09 March 2026 18:44:59 +0000 (0:00:00.074) 0:00:00.154 ********** 2026-03-09 18:44:59,352 p=40838 u=zuul n=ansible | Monday 09 March 2026 18:44:59 +0000 (0:00:00.074) 0:00:00.153 ********** 2026-03-09 18:44:59,488 p=40838 u=zuul n=ansible | ok: [controller] 2026-03-09 18:44:59,499 p=40838 u=zuul n=ansible | TASK [run_hook : Loop on hooks for pre_kuttl_from_operator _raw_params={{ hook.type }}.yml] *** 2026-03-09 18:44:59,499 p=40838 u=zuul n=ansible | Monday 09 March 2026 18:44:59 +0000 (0:00:00.147) 0:00:00.301 ********** 2026-03-09 18:44:59,499 p=40838 u=zuul n=ansible | Monday 09 March 2026 18:44:59 +0000 (0:00:00.147) 0:00:00.301 ********** 2026-03-09 18:44:59,639 p=40838 u=zuul n=ansible | skipping: [controller] 2026-03-09 18:44:59,655 p=40838 u=zuul n=ansible | TASK [run kuttl test suite from operator Makefile chdir={{ ansible_user_dir }}/{{ operator_basedir }}, _raw_params=make kuttl-test-run] *** 2026-03-09 18:44:59,655 p=40838 u=zuul n=ansible | Monday 09 March 2026 18:44:59 +0000 (0:00:00.155) 0:00:00.457 ********** 2026-03-09 18:44:59,655 p=40838 u=zuul n=ansible | Monday 09 March 2026 18:44:59 +0000 (0:00:00.155) 0:00:00.456 ********** 2026-03-09 19:10:03,880 p=40838 u=zuul n=ansible | fatal: [controller]: FAILED! => changed: true cmd: - make - kuttl-test-run delta: '0:25:02.921189' end: '2026-03-09 19:10:02.836855' msg: non-zero return code rc: 2 start: '2026-03-09 18:44:59.915666' stderr: |- Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`. Warning: The 'cinderBackup' field is deprecated and will be removed in a future release. Please migrate to 'cinderBackups'. make: *** [Makefile:444: kuttl-test-run] Error 1 stderr_lines: - 'Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.' - 'Warning: The ''cinderBackup'' field is deprecated and will be removed in a future release. Please migrate to ''cinderBackups''.' - 'make: *** [Makefile:444: kuttl-test-run] Error 1' stdout: "oc kuttl test --v 1 --start-kind=false --config test/kuttl/test-suites/default/config.yaml\n=== RUN kuttl\n harness.go:463: starting setup\n harness.go:255: running tests using configured kubeconfig.\n harness.go:278: Successful connection to cluster at: https://api.crc.testing:6443\n harness.go:363: running tests\n harness.go:75: going to run test suite with timeout of 300 seconds for each step\n harness.go:375: testsuite: test/kuttl/test-suites/default/ has 11 tests\n=== RUN kuttl/harness\n=== RUN kuttl/harness/appcred-tests\n=== PAUSE kuttl/harness/appcred-tests\n=== RUN \ kuttl/harness/common\n=== PAUSE kuttl/harness/common\n=== RUN kuttl/harness/deps\n=== PAUSE kuttl/harness/deps\n=== RUN kuttl/harness/watcher\n=== PAUSE kuttl/harness/watcher\n=== RUN kuttl/harness/watcher-api-scaling\n=== PAUSE kuttl/harness/watcher-api-scaling\n=== RUN kuttl/harness/watcher-cinder\n=== PAUSE kuttl/harness/watcher-cinder\n=== RUN kuttl/harness/watcher-notification\n=== PAUSE kuttl/harness/watcher-notification\n=== RUN kuttl/harness/watcher-rmquser\n=== PAUSE kuttl/harness/watcher-rmquser\n=== RUN kuttl/harness/watcher-tls\n=== PAUSE kuttl/harness/watcher-tls\n=== RUN kuttl/harness/watcher-tls-certs-change\n=== PAUSE kuttl/harness/watcher-tls-certs-change\n=== RUN kuttl/harness/watcher-topology\n=== PAUSE kuttl/harness/watcher-topology\n=== CONT kuttl/harness/appcred-tests\n logger.go:42: 18:45:00 | appcred-tests | Skipping creation of user-supplied namespace: watcher-kuttl-default\n \ logger.go:42: 18:45:00 | appcred-tests/0-cleanup | starting test step 0-cleanup\n \ logger.go:42: 18:45:00 | appcred-tests/0-cleanup | test step completed 0-cleanup\n \ logger.go:42: 18:45:00 | appcred-tests/1-deploy | starting test step 1-deploy\n \ logger.go:42: 18:45:00 | appcred-tests/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 18:45:00 | appcred-tests/1-deploy | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n oc wait -n \"${NS}\" watcher watcher-kuttl --for=condition=Ready --timeout=600s]\n logger.go:42: 18:45:57 | appcred-tests/1-deploy | watcher.watcher.openstack.org/watcher-kuttl condition met\n logger.go:42: 18:45:57 | appcred-tests/1-deploy | test step completed 1-deploy\n \ logger.go:42: 18:45:57 | appcred-tests/2-deploy-appcred | starting test step 2-deploy-appcred\n logger.go:42: 18:45:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ old_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ old_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ old_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n oc create configmap appcred-watcher-pre \\\n \ --from-literal=api_uid=\"${old_api_uid}\" \\\n --from-literal=applier_uid=\"${old_applier_uid}\" \\\n --from-literal=decision_engine_uid=\"${old_de_uid}\" \\\n --dry-run=client -o yaml | oc apply -n \"${NS}\" -f -\n \n cat <<'EOF' | oc apply -n \"${NS}\" -f -\n apiVersion: keystone.openstack.org/v1beta1\n kind: KeystoneApplicationCredential\n metadata:\n name: ac-watcher\n spec:\n \ secret: os**********et\n passwordSelector: WatcherPassword\n userName: watcher\n roles:\n - admin\n - service\n unrestricted: false\n EOF\n \n oc patch watcher watcher-kuttl -n \"${NS}\" --type=merge -p '{\"spec\":{\"auth\":{\"applicationCredentialSecret\":\"ac-watcher-secret\"}}}']\n \ logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | configmap/appcred-watcher-pre created\n logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | keystoneapplicationcredential.keystone.openstack.org/ac-watcher created\n logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:50:58 | appcred-tests/2-deploy-appcred | error: timed out waiting for the condition on keystoneapplicationcredentials/ac-watcher\n logger.go:42: 18:50:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:50:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:50:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:29 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:36 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:44 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:51:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:51:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:51:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:29 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:52:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:52:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:52:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:08 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:15 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:44 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:53:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:53:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:53:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:08 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:15 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:36 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:54:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:54:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:54:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:08 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:55:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:55:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:55:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:08 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:15 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:36 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:56:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:56:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:56:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:08 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:15 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:36 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:44 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:57:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:57:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:57:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:15 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:29 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:36 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:44 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:58:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:58:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:58:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:08 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:15 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:29 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:36 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:44 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 18:59:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 18:59:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 18:59:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:44 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart after appcred secret became available\"\n echo \" api: ${old_api_uid} -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains expected application_credential_id\"]\n logger.go:42: 19:00:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 19:00:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org \"ac-watcher\" not found\n logger.go:42: 19:00:59 | appcred-tests/2-deploy-appcred | test step failed 2-deploy-appcred\n case.go:396: failed in step 2-deploy-appcred\n \ case.go:398: command \"NS=\\\"${NAMESPACE}\\\"\\\\n echo \\\"Waiting for KeystoneApplicationCredential...\" failed, exit status 1\n logger.go:42: 19:00:59 | appcred-tests | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-notification\n logger.go:42: 19:00:59 | watcher-notification | Skipping creation of user-supplied namespace: watcher-kuttl-default\n \ logger.go:42: 19:00:59 | watcher-notification/0-cleanup-watcher | starting test step 0-cleanup-watcher\n logger.go:42: 19:00:59 | watcher-notification/0-cleanup-watcher | test step completed 0-cleanup-watcher\n logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification | starting test step 1-deploy-with-notification\n logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | Now using project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n \ logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | error: Internal error occurred: error executing command in container: container is not created or running\n logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | ++ echo\n logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | + '[' 0 == 1 ']'\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n \ logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | Error from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | ++ echo\n logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | + '[' 0 == 1 ']'\n \ logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | + APIPOD=\n logger.go:42: 19:01:04 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:04 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | + APIPOD=\n logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | + APIPOD=\n logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:08 | watcher-notification/1-deploy-with-notification | + APIPOD=\n logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | + APIPOD=\n logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | + APIPOD=\n logger.go:42: 19:01:11 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:11 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | + APIPOD=\n logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | + APIPOD=\n logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:15 | watcher-notification/1-deploy-with-notification | + APIPOD=\n logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | + APIPOD=\n logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n \ logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:01:18 | watcher-notification/1-deploy-with-notification | error: unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 19:01:18 | watcher-notification/1-deploy-with-notification | ++ echo\n logger.go:42: 19:01:18 | watcher-notification/1-deploy-with-notification | + '[' 0 == 1 ']'\n \ logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n \ logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = messagingv2 transport_url = 'rabbit://**********=1' '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | + '[' 1 == 1 ']'\n logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n \ logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = messagingv2 transport_url = 'rabbit://**********=1' '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | + '[' 1 == 1 ']'\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n \ logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = messagingv2 transport_url = 'rabbit://**********=1' '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | + '[' 1 == 1 ']'\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n \ logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = messagingv2 transport_url = 'rabbit://**********=1' '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | + '[' 1 == 1 ']'\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n \ logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = messagingv2 transport_url = 'rabbit://**********=1' '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | + '[' 1 == 1 ']'\n logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n \ logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = messagingv2 transport_url = 'rabbit://**********=1' '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | + '[' 1 == 1 ']'\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n \ logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ head -1\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ grep -v '^$'\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n \ logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = messagingv2 transport_url = 'rabbit://**********=1' '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | + '[' 1 == 1 ']'\n logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | test step completed 1-deploy-with-notification\n logger.go:42: 19:01:29 | watcher-notification/2-cleanup-watcher | starting test step 2-cleanup-watcher\n logger.go:42: 19:01:35 | watcher-notification/2-cleanup-watcher | test step completed 2-cleanup-watcher\n logger.go:42: 19:01:35 | watcher-notification | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-topology\n \ logger.go:42: 19:01:35 | watcher-topology | Skipping creation of user-supplied namespace: watcher-kuttl-default\n logger.go:42: 19:01:35 | watcher-topology/0-cleanup-watcher | starting test step 0-cleanup-watcher\n logger.go:42: 19:01:35 | watcher-topology/0-cleanup-watcher | test step completed 0-cleanup-watcher\n logger.go:42: 19:01:35 | watcher-topology/1-deploy-with-topology | starting test step 1-deploy-with-topology\n logger.go:42: 19:01:35 | watcher-topology/1-deploy-with-topology | Topology:watcher-kuttl-default/watcher-api created\n logger.go:42: 19:01:35 | watcher-topology/1-deploy-with-topology | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:01:59 | watcher-topology/1-deploy-with-topology | test step completed 1-deploy-with-topology\n logger.go:42: 19:01:59 | watcher-topology/2-cleanup-watcher | starting test step 2-cleanup-watcher\n logger.go:42: 19:02:00 | watcher-topology/2-cleanup-watcher | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 19:02:00 | watcher-topology/2-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:02:00 | watcher-topology/2-cleanup-watcher | ++ grep -c '^watcher'\n logger.go:42: 19:02:02 | watcher-topology/2-cleanup-watcher | + '[' 0 == 0 ']'\n logger.go:42: 19:02:04 | watcher-topology/2-cleanup-watcher | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 19:02:04 | watcher-topology/2-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:02:04 | watcher-topology/2-cleanup-watcher | ++ grep -c '^watcher'\n logger.go:42: 19:02:06 | watcher-topology/2-cleanup-watcher | + '[' 0 == 0 ']'\n logger.go:42: 19:02:07 | watcher-topology/2-cleanup-watcher | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 19:02:07 | watcher-topology/2-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:02:07 | watcher-topology/2-cleanup-watcher | ++ grep -c '^watcher'\n logger.go:42: 19:02:09 | watcher-topology/2-cleanup-watcher | + '[' 0 == 0 ']'\n logger.go:42: 19:02:09 | watcher-topology/2-cleanup-watcher | test step completed 2-cleanup-watcher\n logger.go:42: 19:02:09 | watcher-topology | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-tls-certs-change\n \ logger.go:42: 19:02:09 | watcher-tls-certs-change | Skipping creation of user-supplied namespace: watcher-kuttl-default\n logger.go:42: 19:02:09 | watcher-tls-certs-change/0-cleanup-watcher | starting test step 0-cleanup-watcher\n logger.go:42: 19:02:09 | watcher-tls-certs-change/0-cleanup-watcher | test step completed 0-cleanup-watcher\n logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | starting test step 1-deploy-with-tlse\n logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-internal-svc created\n logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc created\n logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n \ # ensure that the svc secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 19:02:10 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:10 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:11 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 19:02:11 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 19:02:15 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:15 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:16 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 19:02:16 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | error: unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | error: unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 19:02:24 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:24 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:33 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:33 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 1\n fi\n ]\n logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | test step completed 1-deploy-with-tlse\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | starting test step 2-change-public-svc-certificate\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-public-svc --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/dnsNames\", \"value\":['watcher-public.watcher-kuttl-default.svc', 'watcher-public.watcher-kuttl-default.svc.cluster.local']}]'\n ]\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | certificate.cert-manager.io/watcher-public-svc patched\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n \ ]\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | + svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ base64 --decode\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | + secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=\n \ logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | + exit 1\n logger.go:42: 19:02:37 | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n \ ]\n logger.go:42: 19:02:37 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:38 | watcher-tls-certs-change/2-change-public-svc-certificate | error: unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 19:02:38 | watcher-tls-certs-change/2-change-public-svc-certificate | + svc_cert=\n \ logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n \ ]\n logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate | error: unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate | + svc_cert=\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n \ ]\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | + svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ base64 --decode\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | + secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu\n \ logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | test step completed 2-change-public-svc-certificate\n logger.go:42: 19:02:40 | watcher-tls-certs-change/3-change-internal-svc-certificate | starting test step 3-change-internal-svc-certificate\n logger.go:42: 19:02:40 | watcher-tls-certs-change/3-change-internal-svc-certificate | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-internal-svc --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/dnsNames\", \"value\":['watcher-internal.watcher-kuttl-default.svc', 'watcher-internal.watcher-kuttl-default.svc.cluster.local']}]'\n ]\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | certificate.cert-manager.io/watcher-internal-svc patched\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc secret and cert secret match\n \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n \ ]\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | + svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | -----END CERTIFICATE-----'\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | ++ base64 --decode\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | + secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | -----END CERTIFICATE-----'\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+\n \ logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | -----END CERTIFICATE-----' ']'\n logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | test step completed 3-change-internal-svc-certificate\n logger.go:42: 19:02:41 | watcher-tls-certs-change/4-cleanup-watcher | starting test step 4-cleanup-watcher\n \ logger.go:42: 19:02:47 | watcher-tls-certs-change/4-cleanup-watcher | test step completed 4-cleanup-watcher\n logger.go:42: 19:02:47 | watcher-tls-certs-change/5-clenaup-certs | starting test step 5-clenaup-certs\n logger.go:42: 19:02:47 | watcher-tls-certs-change/5-clenaup-certs | test step completed 5-clenaup-certs\n logger.go:42: 19:02:47 | watcher-tls-certs-change | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-tls\n logger.go:42: 19:02:47 | watcher-tls | Skipping creation of user-supplied namespace: watcher-kuttl-default\n \ logger.go:42: 19:02:47 | watcher-tls/0-cleanup-watcher | starting test step 0-cleanup-watcher\n logger.go:42: 19:02:47 | watcher-tls/0-cleanup-watcher | test step completed 0-cleanup-watcher\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | starting test step 1-deploy-with-tlse\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-internal-svc created\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc created\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 2 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | ++ grep -c '^watcher'\n logger.go:42: 19:02:49 | watcher-tls/1-deploy-with-tlse | + '[' 0 == 1 ']'\n logger.go:42: 19:02:50 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 2 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:02:50 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:02:50 | watcher-tls/1-deploy-with-tlse | ++ grep -c '^watcher'\n logger.go:42: 19:02:53 | watcher-tls/1-deploy-with-tlse | + '[' 0 == 1 ']'\n logger.go:42: 19:02:54 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 2 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:02:54 | watcher-tls/1-deploy-with-tlse | ++ grep -c '^watcher'\n logger.go:42: 19:02:54 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | ++ grep watcher\n logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | ++ awk '{print $1}'\n logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | + '[' -n '' ']'\n \ logger.go:42: 19:03:00 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 2 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:03:00 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:03:00 | watcher-tls/1-deploy-with-tlse | ++ grep -c '^watcher'\n logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | ++ grep watcher\n logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | ++ awk '{print $1}'\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n \ logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n \ logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + grep infra-optim\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ grep -c https\n logger.go:42: 19:03:06 | watcher-tls/1-deploy-with-tlse | + '[' 0 == 2 ']'\n logger.go:42: 19:03:07 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 2 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:03:07 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:03:07 | watcher-tls/1-deploy-with-tlse | ++ grep -c '^watcher'\n logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | ++ awk '{print $1}'\n logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | ++ grep watcher\n logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:03:11 | watcher-tls/1-deploy-with-tlse | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n \ logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n \ logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + grep infra-optim\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ grep -c https\n logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n \ logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | + '[' '' == '' ']'\n \ logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | + exit 0\n logger.go:42: 19:03:16 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n \ oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 2 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:03:16 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:03:16 | watcher-tls/1-deploy-with-tlse | ++ grep -c '^watcher'\n logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse | ++ awk '{print $1}'\n logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse | ++ grep watcher\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n \ logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + grep infra-optim\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ grep -c https\n logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n \ logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | + '[' '' == '' ']'\n \ logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | + exit 0\n logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | test step completed 1-deploy-with-tlse\n \ logger.go:42: 19:03:22 | watcher-tls/2-patch-mtls | starting test step 2-patch-mtls\n \ logger.go:42: 19:03:22 | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail\n oc patch oscp -n $NAMESPACE openstack --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode\", \"value\": \"Request\"}]'\n ]\n logger.go:42: 19:03:22 | watcher-tls/2-patch-mtls | + oc patch oscp -n watcher-kuttl-default openstack --type=json '-p=[{\"op\": \"replace\", \"path\": \"/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode\", \"value\": \"Request\"}]'\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | openstackcontrolplane.core.openstack.org/openstack patched\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n # Get pod names for each watcher service\n APIPOD=$(oc get pods -l service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n \n \ # Verify memcached mTLS config parameters in watcher-api config\n if [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n \ exit 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n \ exit 1\n fi\n ]\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n \ logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n \ logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n \ logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n \ logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + '[' 0 == 1 ']'\n logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n # Get pod names for each watcher service\n APIPOD=$(oc get pods -l service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n \n \ # Verify memcached mTLS config parameters in watcher-api config\n if [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n \ exit 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n \ exit 1\n fi\n ]\n logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default\n logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n \ logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n \ logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n \ logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n \ logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | error: cannot exec into a container in a completed pod; current phase is Failed\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ echo\n logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + '[' 0 == 1 ']'\n logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n \ # Get pod names for each watcher service\n APIPOD=$(oc get pods -l service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n \ \n # Verify memcached mTLS config parameters in watcher-api config\n \ if [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n \ exit 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n \ exit 1\n fi\n ]\n logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default\n logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n \ logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n \ logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n \ logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n \ logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | error: unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | ++ echo\n logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | + '[' 0 == 1 ']'\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n \ # Get pod names for each watcher service\n APIPOD=$(oc get pods -l service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n \ \n # Verify memcached mTLS config parameters in watcher-api config\n \ if [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n \ exit 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n \ exit 1\n fi\n ]\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n \ logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n \ logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n \ logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n \ logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile = /etc/pki/tls/private/mtls.key'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled = true'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n \ logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n \ logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n \ logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-applier-0 ']'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-applier config...'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | Checking watcher-applier config...\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile = /etc/pki/tls/private/mtls.key'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled = true'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n \ logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n \ logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n \ logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-decision-engine-0 ']'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-decision-engine config...'\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine config...\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:32 | watcher-tls/2-patch-mtls | error: Internal error occurred: error executing command in container: container is not created or running\n logger.go:42: 19:03:32 | watcher-tls/2-patch-mtls | ++ echo\n logger.go:42: 19:03:32 | watcher-tls/2-patch-mtls | + '[' 0 == 1 ']'\n logger.go:42: 19:03:33 | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n \ # Get pod names for each watcher service\n APIPOD=$(oc get pods -l service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n \ \n # Verify memcached mTLS config parameters in watcher-api config\n \ if [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n \ exit 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n \ exit 1\n fi\n ]\n logger.go:42: 19:03:33 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n \ logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n \ logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n \ logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n \ logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile = /etc/pki/tls/private/mtls.key'\n logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled = true'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n \ logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n \ logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n \ logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-applier-0 ']'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-applier config...'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | Checking watcher-applier config...\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile = /etc/pki/tls/private/mtls.key'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled = true'\n logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n \ logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n \ logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n \ logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-decision-engine-0 ']'\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-decision-engine config...'\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine config...\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile = /etc/pki/tls/private/mtls.key'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled = true'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n \ logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n \ logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | test step completed 2-patch-mtls\n logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | starting test step 3-disable-podlevel-tls\n logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/tls/api\", \"value\":{ \"internal\": {}, \"public\": {} }}]'\n ]\n logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n \ oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # check that watcher internal endpoint does not use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]\n # check that watcher public endpoint does use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep public | [ $(grep -c https) == 1 ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n \ counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n \ ]\n logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | ++ grep -c '^watcher'\n logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher\n logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls | ++ awk '{print $1}'\n logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:03:43 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:03:43 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + grep internal\n logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n logger.go:42: 19:03:46 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 0 ']'\n logger.go:42: 19:03:47 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n \ # check that watcher internal endpoint does not use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]\n # check that watcher public endpoint does use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep public | [ $(grep -c https) == 1 ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:03:47 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:03:47 | watcher-tls/3-disable-podlevel-tls | ++ grep -c '^watcher'\n logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher\n logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls | ++ awk '{print $1}'\n logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + grep internal\n \ logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n \ logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n logger.go:42: 19:03:53 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 0 ']'\n logger.go:42: 19:03:54 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # check that watcher internal endpoint does not use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]\n \ # check that watcher public endpoint does use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep public | [ $(grep -c https) == 1 ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n \ fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n \ counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n \ ]\n logger.go:42: 19:03:54 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:03:54 | watcher-tls/3-disable-podlevel-tls | ++ grep -c '^watcher'\n logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher\n logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls | ++ awk '{print $1}'\n logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + grep internal\n logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n logger.go:42: 19:04:00 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 0 ']'\n logger.go:42: 19:04:01 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n \ # check that watcher internal endpoint does not use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]\n # check that watcher public endpoint does use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep public | [ $(grep -c https) == 1 ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:04:01 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:04:01 | watcher-tls/3-disable-podlevel-tls | ++ grep -c '^watcher'\n logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls | ++ awk '{print $1}'\n logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher\n logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + grep internal\n \ logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n \ logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n \ logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | + '[' 0 == 0 ']'\n logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | + grep public\n logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n \ logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n \ logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | + '[' '' == '' ']'\n logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | + exit 0\n logger.go:42: 19:04:11 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n \ oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # check that watcher internal endpoint does not use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]\n # check that watcher public endpoint does use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep public | [ $(grep -c https) == 1 ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n \ counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n \ ]\n logger.go:42: 19:04:11 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:04:11 | watcher-tls/3-disable-podlevel-tls | ++ grep -c '^watcher'\n logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher\n logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls | ++ awk '{print $1}'\n logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + grep internal\n \ logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n \ logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | + '[' 0 == 0 ']'\n logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | + grep public\n logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n \ logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n \ logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | + '[' '' == '' ']'\n logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | + exit 0\n logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | test step completed 3-disable-podlevel-tls\n \ logger.go:42: 19:04:20 | watcher-tls/4-deploy-without-route | starting test step 4-deploy-without-route\n logger.go:42: 19:04:20 | watcher-tls/4-deploy-without-route | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/override\", \"value\":{\"service\": { \"internal\": {}, \"public\": { \"metadata\": { \"annotations\": { \"metallb.universe.tf/address-pool\": \"ctlplane\", \"metallb.universe.tf/allow-shared-ip\": \"ctlplane\" } }, \"spec\": { \"type\": \"LoadBalancer\" } } } }}]'\n ]\n logger.go:42: 19:04:20 | watcher-tls/4-deploy-without-route | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 19:04:21 | watcher-tls/4-deploy-without-route | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n \ counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n \ ]\n logger.go:42: 19:04:21 | watcher-tls/4-deploy-without-route | ++ grep -c '^watcher'\n logger.go:42: 19:04:21 | watcher-tls/4-deploy-without-route | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route | + '[' 1 == 1 ']'\n logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route | ++ awk '{print $1}'\n logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route | ++ grep watcher\n logger.go:42: 19:04:24 | watcher-tls/4-deploy-without-route | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:04:24 | watcher-tls/4-deploy-without-route | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + '[' '' == '' ']'\n logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + exit 0\n logger.go:42: 19:04:26 | watcher-tls/4-deploy-without-route | running command: [sh -c set -euxo pipefail\n \ oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:04:26 | watcher-tls/4-deploy-without-route | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:04:26 | watcher-tls/4-deploy-without-route | ++ grep -c '^watcher'\n logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route | + '[' 1 == 1 ']'\n logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route | ++ awk '{print $1}'\n logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route | ++ grep watcher\n logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + '[' '' == '' ']'\n logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + exit 0\n logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | test step completed 4-deploy-without-route\n \ logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | starting test step 5-disable-tls\n \ logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/override\", \"value\":{}}]'\n ]\n logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n # check that no watcher endpoint uses https\n \ oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 0 ]\n ]\n logger.go:42: 19:04:31 | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:04:31 | watcher-tls/5-disable-tls | ++ grep -c '^watcher'\n logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | + '[' 1 == 1 ']'\n logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | ++ grep watcher\n logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | ++ awk '{print $1}'\n logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n \ logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + '[' '' == '' ']'\n logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + exit 0\n logger.go:42: 19:04:36 | watcher-tls/5-disable-tls | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n \ counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n \ # check that no watcher endpoint uses https\n oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 0 ]\n ]\n logger.go:42: 19:04:36 | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:04:36 | watcher-tls/5-disable-tls | ++ grep -c '^watcher'\n logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | + '[' 1 == 1 ']'\n logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | ++ grep watcher\n \ logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | ++ awk '{print $1}'\n logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3\n logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + '[' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 ']'\n \ logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + '[' '' == '' ']'\n logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + exit 0\n logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | test step completed 5-disable-tls\n logger.go:42: 19:04:40 | watcher-tls/6-cleanup-watcher | starting test step 6-cleanup-watcher\n logger.go:42: 19:04:41 | watcher-tls/6-cleanup-watcher | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 19:04:41 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:04:41 | watcher-tls/6-cleanup-watcher | ++ grep -c '^watcher'\n logger.go:42: 19:04:43 | watcher-tls/6-cleanup-watcher | + '[' 1 == 0 ']'\n logger.go:42: 19:04:44 | watcher-tls/6-cleanup-watcher | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 19:04:44 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:04:44 | watcher-tls/6-cleanup-watcher | ++ grep -c '^watcher'\n logger.go:42: 19:04:47 | watcher-tls/6-cleanup-watcher | + '[' 0 == 0 ']'\n logger.go:42: 19:04:47 | watcher-tls/6-cleanup-watcher | test step completed 6-cleanup-watcher\n logger.go:42: 19:04:47 | watcher-tls/7-cleanup-certs | starting test step 7-cleanup-certs\n logger.go:42: 19:04:47 | watcher-tls/7-cleanup-certs | test step completed 7-cleanup-certs\n logger.go:42: 19:04:47 | watcher-tls | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-rmquser\n logger.go:42: 19:04:47 | watcher-rmquser | Skipping creation of user-supplied namespace: watcher-kuttl-default\n \ logger.go:42: 19:04:47 | watcher-rmquser/0-cleanup-watcher | starting test step 0-cleanup-watcher\n logger.go:42: 19:04:47 | watcher-rmquser/0-cleanup-watcher | test step completed 0-cleanup-watcher\n logger.go:42: 19:04:47 | watcher-rmquser/1-deploy | starting test step 1-deploy\n logger.go:42: 19:04:47 | watcher-rmquser/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:04:47 | watcher-rmquser/1-deploy | running command: [sh -c set -euxo pipefail\n \n \ # Wait for Watcher to be Ready\n kubectl wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE --timeout=300s\n \n # Verify WatcherNotificationTransportURLReady condition exists and is True\n kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath='{.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}' | grep -q \"True\"\n echo \"WatcherNotificationTransportURLReady condition is True\"\n \n # Count TransportURL CRs - should be exactly 2 (one for messaging, one for notifications)\n transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-transport\" | wc -l)\n notification_transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-notification\" | wc -l)\n \n if [ \"$transport_count\" -ne \"1\" ]; then\n echo \"Expected 1 watcher-transport TransportURL, found $transport_count\"\n exit 1\n fi\n \n if [ \"$notification_transport_count\" -ne \"1\" ]; then\n echo \"Expected 1 notification-transport TransportURL, found $notification_transport_count\"\n exit 1\n fi\n \n echo \"Correctly found 2 TransportURLs (separate clusters: transport and notification)\"\n \ \n # Verify watcher-transport has correct user and vhost\n transport_user=$(kubectl get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath='{.spec.username}')\n \ transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n if [ \"$transport_user\" != \"watcher-rpc\" ]; then\n echo \"Expected watcher-transport username 'watcher-rpc', found '$transport_user'\"\n exit 1\n fi\n if [ \"$transport_vhost\" != \"watcher-rpc\" ]; then\n echo \"Expected watcher-transport vhost 'watcher-rpc', found '$transport_vhost'\"\n exit 1\n fi\n echo \"Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)\"\n \n # Verify notification-transport has correct user and vhost\n notif_user=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath='{.spec.username}')\n notif_vhost=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n \ if [ \"$notif_user\" != \"watcher-notifications\" ]; then\n echo \"Expected notification-transport username 'watcher-notifications', found '$notif_user'\"\n \ exit 1\n fi\n if [ \"$notif_vhost\" != \"watcher-notifications\" ]; then\n echo \"Expected notification-transport vhost 'watcher-notifications', found '$notif_vhost'\"\n exit 1\n fi\n echo \"Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)\"\n \ \n # Verify that watcher.conf contains the notifications transport_url\n \ WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l \"service=watcher-api\" -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)\n if [ -z \"${WATCHER_API_POD}\" ]; then\n echo \"No watcher-api pod found\"\n \ exit 1\n fi\n # Verify RPC transport_url in DEFAULT section\n \ rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E '^\\[DEFAULT\\]' -A 50 | grep 'transport_url' | head -1 || true)\n if [ -z \"$rpc_transport_url\" ]; then\n echo \"transport_url not found in DEFAULT section\"\n exit 1\n fi\n echo \"Found RPC transport_url: $rpc_transport_url\"\n \n \ # Verify the RPC transport_url contains the correct vhost (watcher-rpc)\n \ if ! echo \"$rpc_transport_url\" | grep -q '/watcher-rpc'; then\n echo \"RPC transport_url does not contain expected vhost '/watcher-rpc'\"\n exit 1\n fi\n echo \"Successfully verified vhost 'watcher-rpc' in RPC transport_url\"\n \ \n # Verify the RPC transport_url contains the correct username (watcher-rpc)\n \ if ! echo \"$rpc_transport_url\" | grep -q 'watcher-rpc:'; then\n echo \"RPC transport_url does not contain expected username 'watcher-rpc:'\"\n exit 1\n fi\n echo \"Successfully verified username 'watcher-rpc' in RPC transport_url\"\n \n # Verify oslo_messaging_notifications section has transport_url configured\n notif_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -A 5 '\\[oslo_messaging_notifications\\]' | grep 'transport_url' || true)\n \ if [ -z \"$notif_transport_url\" ]; then\n echo \"transport_url not found in oslo_messaging_notifications section\"\n exit 1\n fi\n \ echo \"Found notifications transport_url: $notif_transport_url\"\n \n \ # Verify the notifications transport_url contains the correct vhost (watcher-notifications)\n \ if ! echo \"$notif_transport_url\" | grep -q '/watcher-notifications'; then\n \ echo \"Notifications transport_url does not contain expected vhost '/watcher-notifications'\"\n \ exit 1\n fi\n echo \"Successfully verified vhost 'watcher-notifications' in notifications transport_url\"\n \n # Verify the notifications transport_url contains the correct username (watcher-notifications)\n if ! echo \"$notif_transport_url\" | grep -q 'watcher-notifications:'; then\n echo \"Notifications transport_url does not contain expected username 'watcher-notifications:'\"\n exit 1\n \ fi\n echo \"Successfully verified username 'watcher-notifications' in notifications transport_url\"\n \n exit 0\n ]\n logger.go:42: 19:04:47 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready watcher/watcher-kuttl -n watcher-kuttl-default --timeout=300s\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | watcher.watcher.openstack.org/watcher-kuttl condition met\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl -n watcher-kuttl-default -o 'jsonpath={.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'\n \ logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + grep -q True\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo 'WatcherNotificationTransportURLReady condition is True'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady condition is True\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl -n watcher-kuttl-default -o name\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ wc -l\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + transport_count=1\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl -n watcher-kuttl-default -o name\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ wc -l\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + notification_transport_count=1\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + '[' 1 -ne 1 ']'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + '[' 1 -ne 1 ']'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo 'Correctly found 2 TransportURLs (separate clusters: transport and notification)'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs (separate clusters: transport and notification)\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default -o 'jsonpath={.spec.username}'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + transport_user=watcher-rpc\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default -o 'jsonpath={.spec.vhost}'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + transport_vhost=watcher-rpc\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + '[' watcher-rpc '!=' watcher-rpc ']'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + '[' watcher-rpc '!=' watcher-rpc ']'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo 'Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)'\n \ logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default -o 'jsonpath={.spec.username}'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + notif_user=watcher-notifications\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default -o 'jsonpath={.spec.vhost}'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + '[' watcher-notifications '!=' watcher-notifications ']'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + '[' watcher-notifications '!=' watcher-notifications ']'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo 'Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)'\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)\n \ logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get pods -n watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name --no-headers\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ head -1\n logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ grep -v '^$'\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + '[' -z watcher-kuttl-api-0 ']'\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep -E '^\\[DEFAULT\\]' -A 50\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep transport_url\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ head -1\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + rpc_transport_url='transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + '[' -z 'transport_url = rabbit://**********=1' ']'\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'Found RPC transport_url: transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Found RPC transport_url: transport_url = rabbit://**********=1\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'Successfully verified vhost '\\''watcher-rpc'\\'' in RPC transport_url'\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-rpc' in RPC transport_url\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'Successfully verified username '\\''watcher-rpc'\\'' in RPC transport_url'\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Successfully verified username 'watcher-rpc' in RPC transport_url\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep -A 5 '\\[oslo_messaging_notifications\\]'\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep transport_url\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + notif_transport_url='transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + '[' -z 'transport_url = rabbit://**********=1' ']'\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'Found notifications transport_url: transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Found notifications transport_url: transport_url = rabbit://**********=1\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'Successfully verified vhost '\\''watcher-notifications'\\'' in notifications transport_url'\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-notifications' in notifications transport_url\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:\n \ logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo 'Successfully verified username '\\''watcher-notifications'\\'' in notifications transport_url'\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Successfully verified username 'watcher-notifications' in notifications transport_url\n logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + exit 0\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | running command: [sh -c set -euxo pipefail\n \n # Wait for Watcher to be Ready\n kubectl wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE --timeout=300s\n \ \n # Verify WatcherNotificationTransportURLReady condition exists and is True\n kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath='{.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}' | grep -q \"True\"\n echo \"WatcherNotificationTransportURLReady condition is True\"\n \n # Count TransportURL CRs - should be exactly 2 (one for messaging, one for notifications)\n transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-transport\" | wc -l)\n notification_transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-notification\" | wc -l)\n \n if [ \"$transport_count\" -ne \"1\" ]; then\n echo \"Expected 1 watcher-transport TransportURL, found $transport_count\"\n exit 1\n fi\n \n if [ \"$notification_transport_count\" -ne \"1\" ]; then\n echo \"Expected 1 notification-transport TransportURL, found $notification_transport_count\"\n exit 1\n fi\n \n echo \"Correctly found 2 TransportURLs (separate clusters: transport and notification)\"\n \ \n # Verify watcher-transport has correct user and vhost\n transport_user=$(kubectl get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath='{.spec.username}')\n \ transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n if [ \"$transport_user\" != \"watcher-rpc\" ]; then\n echo \"Expected watcher-transport username 'watcher-rpc', found '$transport_user'\"\n exit 1\n fi\n if [ \"$transport_vhost\" != \"watcher-rpc\" ]; then\n echo \"Expected watcher-transport vhost 'watcher-rpc', found '$transport_vhost'\"\n exit 1\n fi\n echo \"Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)\"\n \n # Verify notification-transport has correct user and vhost\n notif_user=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath='{.spec.username}')\n notif_vhost=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n \ if [ \"$notif_user\" != \"watcher-notifications\" ]; then\n echo \"Expected notification-transport username 'watcher-notifications', found '$notif_user'\"\n \ exit 1\n fi\n if [ \"$notif_vhost\" != \"watcher-notifications\" ]; then\n echo \"Expected notification-transport vhost 'watcher-notifications', found '$notif_vhost'\"\n exit 1\n fi\n echo \"Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)\"\n \ \n # Verify that watcher.conf contains the notifications transport_url\n \ WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l \"service=watcher-api\" -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)\n if [ -z \"${WATCHER_API_POD}\" ]; then\n echo \"No watcher-api pod found\"\n \ exit 1\n fi\n # Verify RPC transport_url in DEFAULT section\n \ rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E '^\\[DEFAULT\\]' -A 50 | grep 'transport_url' | head -1 || true)\n if [ -z \"$rpc_transport_url\" ]; then\n echo \"transport_url not found in DEFAULT section\"\n exit 1\n fi\n echo \"Found RPC transport_url: $rpc_transport_url\"\n \n \ # Verify the RPC transport_url contains the correct vhost (watcher-rpc)\n \ if ! echo \"$rpc_transport_url\" | grep -q '/watcher-rpc'; then\n echo \"RPC transport_url does not contain expected vhost '/watcher-rpc'\"\n exit 1\n fi\n echo \"Successfully verified vhost 'watcher-rpc' in RPC transport_url\"\n \ \n # Verify the RPC transport_url contains the correct username (watcher-rpc)\n \ if ! echo \"$rpc_transport_url\" | grep -q 'watcher-rpc:'; then\n echo \"RPC transport_url does not contain expected username 'watcher-rpc:'\"\n exit 1\n fi\n echo \"Successfully verified username 'watcher-rpc' in RPC transport_url\"\n \n # Verify oslo_messaging_notifications section has transport_url configured\n notif_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -A 5 '\\[oslo_messaging_notifications\\]' | grep 'transport_url' || true)\n \ if [ -z \"$notif_transport_url\" ]; then\n echo \"transport_url not found in oslo_messaging_notifications section\"\n exit 1\n fi\n \ echo \"Found notifications transport_url: $notif_transport_url\"\n \n \ # Verify the notifications transport_url contains the correct vhost (watcher-notifications)\n \ if ! echo \"$notif_transport_url\" | grep -q '/watcher-notifications'; then\n \ echo \"Notifications transport_url does not contain expected vhost '/watcher-notifications'\"\n \ exit 1\n fi\n echo \"Successfully verified vhost 'watcher-notifications' in notifications transport_url\"\n \n # Verify the notifications transport_url contains the correct username (watcher-notifications)\n if ! echo \"$notif_transport_url\" | grep -q 'watcher-notifications:'; then\n echo \"Notifications transport_url does not contain expected username 'watcher-notifications:'\"\n exit 1\n \ fi\n echo \"Successfully verified username 'watcher-notifications' in notifications transport_url\"\n \n exit 0\n ]\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready watcher/watcher-kuttl -n watcher-kuttl-default --timeout=300s\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | watcher.watcher.openstack.org/watcher-kuttl condition met\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl -n watcher-kuttl-default -o 'jsonpath={.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'\n \ logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + grep -q True\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + echo 'WatcherNotificationTransportURLReady condition is True'\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady condition is True\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | ++ kubectl get transporturl -n watcher-kuttl-default -o name\n logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | ++ wc -l\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + transport_count=1\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ wc -l\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification\n \ logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl -n watcher-kuttl-default -o name\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + notification_transport_count=1\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + '[' 1 -ne 1 ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + '[' 1 -ne 1 ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + echo 'Correctly found 2 TransportURLs (separate clusters: transport and notification)'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs (separate clusters: transport and notification)\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default -o 'jsonpath={.spec.username}'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + transport_user=watcher-rpc\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default -o 'jsonpath={.spec.vhost}'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + transport_vhost=watcher-rpc\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + '[' watcher-rpc '!=' watcher-rpc ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + '[' watcher-rpc '!=' watcher-rpc ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + echo 'Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)'\n \ logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default -o 'jsonpath={.spec.username}'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + notif_user=watcher-notifications\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default -o 'jsonpath={.spec.vhost}'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + '[' watcher-notifications '!=' watcher-notifications ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + '[' watcher-notifications '!=' watcher-notifications ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + echo 'Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)\n \ logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep -v '^$'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ head -1\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get pods -n watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name --no-headers\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0\n \ logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + '[' -z watcher-kuttl-api-0 ']'\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep -E '^\\[DEFAULT\\]' -A 50\n logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep transport_url\n \ logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ head -1\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + rpc_transport_url='transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + '[' -z 'transport_url = rabbit://**********=1' ']'\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'Found RPC transport_url: transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Found RPC transport_url: transport_url = rabbit://**********=1\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'Successfully verified vhost '\\''watcher-rpc'\\'' in RPC transport_url'\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-rpc' in RPC transport_url\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'Successfully verified username '\\''watcher-rpc'\\'' in RPC transport_url'\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Successfully verified username 'watcher-rpc' in RPC transport_url\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | ++ grep transport_url\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | ++ grep -A 5 '\\[oslo_messaging_notifications\\]'\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + notif_transport_url='transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + '[' -z 'transport_url = rabbit://**********=1' ']'\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'Found notifications transport_url: transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Found notifications transport_url: transport_url = rabbit://**********=1\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'Successfully verified vhost '\\''watcher-notifications'\\'' in notifications transport_url'\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-notifications' in notifications transport_url\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'transport_url = rabbit://**********=1'\n \ logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo 'Successfully verified username '\\''watcher-notifications'\\'' in notifications transport_url'\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Successfully verified username 'watcher-notifications' in notifications transport_url\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + exit 0\n logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | test step completed 1-deploy\n logger.go:42: 19:05:16 | watcher-rmquser/2-cleanup-watcher | starting test step 2-cleanup-watcher\n logger.go:42: 19:05:22 | watcher-rmquser/2-cleanup-watcher | test step completed 2-cleanup-watcher\n logger.go:42: 19:05:22 | watcher-rmquser | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher\n logger.go:42: 19:05:22 | watcher | Skipping creation of user-supplied namespace: watcher-kuttl-default\n \ logger.go:42: 19:05:22 | watcher/0-cleanup-watcher | starting test step 0-cleanup-watcher\n \ logger.go:42: 19:05:22 | watcher/0-cleanup-watcher | test step completed 0-cleanup-watcher\n \ logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | starting test step 1-deploy-with-defaults\n logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n \ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" -ge 1 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:05:24 | watcher/1-deploy-with-defaults | + '[' 0 == 1 ']'\n logger.go:42: 19:05:25 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" -ge 1 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:05:25 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:05:25 | watcher/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:05:28 | watcher/1-deploy-with-defaults | + '[' 0 == 1 ']'\n logger.go:42: 19:05:29 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" -ge 1 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:05:29 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:05:29 | watcher/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | ++ grep watcher\n logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | ++ awk '{print $1}'\n logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136\n logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | + '[' 5be153439946473bb5b269e3a0a3e136 == 5be153439946473bb5b269e3a0a3e136 ']'\n logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | + '[' -n '' ']'\n \ logger.go:42: 19:05:34 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n \ SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n \ [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" -ge 1 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:05:34 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:05:34 | watcher/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | ++ awk '{print $1}'\n logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | ++ grep watcher\n logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + '[' 5be153439946473bb5b269e3a0a3e136 == 5be153439946473bb5b269e3a0a3e136 ']'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ base64 -d\n \ logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ grep -c 'region_name = regionOne'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + '[' 9 -ge 1 ']'\n logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n \ logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n \ logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + exit 0\n logger.go:42: 19:05:41 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n \ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" -ge 1 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:05:41 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:05:41 | watcher/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | ++ awk '{print $1}'\n logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | ++ grep watcher\n logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + '[' 5be153439946473bb5b269e3a0a3e136 == 5be153439946473bb5b269e3a0a3e136 ']'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ base64 -d\n \ logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n \ logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ grep -c 'region_name = regionOne'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + '[' 9 -ge 1 ']'\n logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 19:05:46 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n logger.go:42: 19:05:46 | watcher/1-deploy-with-defaults | + exit 0\n logger.go:42: 19:05:47 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n \ SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n \ [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" -ge 1 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:05:47 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:05:47 | watcher/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults | ++ awk '{print $1}'\n logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults | ++ grep watcher\n logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136\n logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + '[' 5be153439946473bb5b269e3a0a3e136 == 5be153439946473bb5b269e3a0a3e136 ']'\n logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n \ logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n \ logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | ++ grep -c 'region_name = regionOne'\n logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + '[' 9 -ge 1 ']'\n logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n \ logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n \ logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + exit 0\n logger.go:42: 19:05:53 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n \ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" -ge 1 ]\n # If we are running the container locally, skip following test\n \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:05:53 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:05:53 | watcher/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | ++ awk '{print $1}'\n logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | ++ grep watcher\n logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136\n logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | + '[' 5be153439946473bb5b269e3a0a3e136 == 5be153439946473bb5b269e3a0a3e136 ']'\n logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n \ logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ base64 -d\n \ logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n \ logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n \ logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ grep -c 'region_name = regionOne'\n \ logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + '[' 9 -ge 1 ']'\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + exit 0\n logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | test step completed 1-deploy-with-defaults\n logger.go:42: 19:05:58 | watcher/2-cleanup-watcher | starting test step 2-cleanup-watcher\n logger.go:42: 19:05:58 | watcher/2-cleanup-watcher | test step completed 2-cleanup-watcher\n logger.go:42: 19:05:58 | watcher/3-precreate-mariadbaccount | starting test step 3-precreate-mariadbaccount\n logger.go:42: 19:05:58 | watcher/3-precreate-mariadbaccount | MariaDBAccount:watcher-kuttl-default/watcher-precreated created\n logger.go:42: 19:05:58 | watcher/3-precreate-mariadbaccount | test step completed 3-precreate-mariadbaccount\n \ logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | starting test step 4-deploy-with-precreated-account\n logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | Secret:wa**********ig created\n logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | error: Internal error occurred: error executing command in container: container is not created or running\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ echo\n logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | + '[' 0 == 1 ']'\n logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:01 | watcher/4-deploy-with-precreated-account | + APIPOD=\n logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | + APIPOD=\n logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | + APIPOD=\n logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:05 | watcher/4-deploy-with-precreated-account | + APIPOD=\n logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | + APIPOD=\n logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | + APIPOD=\n logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:09 | watcher/4-deploy-with-precreated-account | + APIPOD=\n logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | + APIPOD=\n logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | + APIPOD=\n logger.go:42: 19:06:12 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:12 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | + APIPOD=\n logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | + APIPOD=\n logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | error: unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | ++ echo\n logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | + '[' 0 == 1 ']'\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:21 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:21 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:23 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:23 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n \ logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:32 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:32 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:41 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:41 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n \ logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:07:04 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:07:04 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n \ logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default\n logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n \ logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n \ logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | ++ head -1\n logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | ++ grep -v '^$'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Global config'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n \ logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo '#' Global config\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ grep -c '^# Service config'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n \ logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo '#' Service config\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n \ logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n \ logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://**********=1' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log '#' empty notification_level means that no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' interface = internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' period = 900\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n \ logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc configuration '' ServerName watcher-public.watcher-kuttl-default.svc '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | test step completed 4-deploy-with-precreated-account\n \ logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | starting test step 5-cleanup-watcher\n \ logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n logger.go:42: 19:07:11 | watcher/5-cleanup-watcher | + '[' 1 == 0 ']'\n logger.go:42: 19:07:14 | watcher/5-cleanup-watcher | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 19:07:14 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:07:14 | watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n \ logger.go:42: 19:07:16 | watcher/5-cleanup-watcher | + '[' 0 == 0 ']'\n logger.go:42: 19:07:16 | watcher/5-cleanup-watcher | test step completed 5-cleanup-watcher\n logger.go:42: 19:07:16 | watcher | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-cinder\n \ logger.go:42: 19:07:16 | watcher-cinder | Skipping creation of user-supplied namespace: watcher-kuttl-default\n logger.go:42: 19:07:16 | watcher-cinder/0-cleanup-watcher | starting test step 0-cleanup-watcher\n logger.go:42: 19:07:16 | watcher-cinder/0-cleanup-watcher | test step completed 0-cleanup-watcher\n logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | starting test step 1-deploy-watcher-no-cinder\n logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n \ logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:19 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:19 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:19 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:20 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:20 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" is waiting to start: ContainerCreating\n logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" is waiting to start: ContainerCreating\n logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 == 2 ']'\n logger.go:42: 19:07:33 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:34 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 == 2 ']'\n logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 == 2 ']'\n logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 == 2 ']'\n logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 == 2 ']'\n logger.go:42: 19:07:38 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:39 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 == 2 ']'\n logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 == 2 ']'\n logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | test step completed 1-deploy-watcher-no-cinder\n logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | starting test step 2-deploy-cinder\n logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | OpenStackControlPlane:watcher-kuttl-default/openstack updated\n logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n \ # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:43 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:43 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:43 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:44 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:48 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:48 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:48 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:49 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:54 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:54 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:54 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:55 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:00 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:00 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:00 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:01 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:06 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:06 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:06 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:07 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:12 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:12 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:12 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:13 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:17 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:17 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:17 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:18 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:23 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:23 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:23 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:24 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | + '[' 2 == 0 ']'\n logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" is waiting to start: ContainerCreating\n logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | + '[' 0 == 0 ']'\n logger.go:42: 19:08:29 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:29 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:29 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:30 | watcher-cinder/2-deploy-cinder | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" is waiting to start: ContainerCreating\n logger.go:42: 19:08:30 | watcher-cinder/2-deploy-cinder | + '[' 0 == 0 ']'\n logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | + '[' 0 == 0 ']'\n logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | + '[' 0 == 0 ']'\n logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | + '[' 0 == 0 ']'\n logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | + '[' 0 == 0 ']'\n logger.go:42: 19:08:35 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:35 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:35 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:36 | watcher-cinder/2-deploy-cinder | + '[' 0 == 0 ']'\n logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | + '[' 0 == 0 ']'\n logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | + '[' 0 == 0 ']'\n logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | + '[' 0 == 0 ']'\n logger.go:42: 19:08:40 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision detects that there is a cinder service and\n # does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 19:08:40 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:40 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:41 | watcher-cinder/2-deploy-cinder | + '[' 0 == 0 ']'\n logger.go:42: 19:08:41 | watcher-cinder/2-deploy-cinder | test step completed 2-deploy-cinder\n logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | starting test step 3-remove-cinder\n logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | OpenStackControlPlane:watcher-kuttl-default/openstack updated\n logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n \ # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n \ ]\n logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n \ logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:08:44 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:44 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:44 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:45 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n \ # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n \ ]\n logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" is waiting to start: ContainerCreating\n logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" is waiting to start: ContainerCreating\n logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n \ logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n logger.go:42: 19:08:52 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:52 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:52 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:08:53 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n \ logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n \ logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n \ logger.go:42: 19:08:58 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:08:58 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:08:58 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 19:08:59 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n # check that the decision engine correctly detects that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n \ logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n \ logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | test step completed 3-remove-cinder\n logger.go:42: 19:09:00 | watcher-cinder/4-cleanup-watcher | starting test step 4-cleanup-watcher\n logger.go:42: 19:09:08 | watcher-cinder/4-cleanup-watcher | test step completed 4-cleanup-watcher\n logger.go:42: 19:09:08 | watcher-cinder | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-api-scaling\n \ logger.go:42: 19:09:08 | watcher-api-scaling | Skipping creation of user-supplied namespace: watcher-kuttl-default\n logger.go:42: 19:09:08 | watcher-api-scaling/0-cleanup-watcher | starting test step 0-cleanup-watcher\n logger.go:42: 19:09:08 | watcher-api-scaling/0-cleanup-watcher | test step completed 0-cleanup-watcher\n logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | starting test step 1-deploy-with-defaults\n logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n ]\n logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:09:11 | watcher-api-scaling/1-deploy-with-defaults | + '[' 0 == 1 ']'\n logger.go:42: 19:09:12 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n \ counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n \ ]\n logger.go:42: 19:09:12 | watcher-api-scaling/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:09:12 | watcher-api-scaling/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:09:14 | watcher-api-scaling/1-deploy-with-defaults | + '[' 0 == 1 ']'\n logger.go:42: 19:09:15 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n \ counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n \ ]\n logger.go:42: 19:09:15 | watcher-api-scaling/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:09:15 | watcher-api-scaling/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults | ++ awk '{print $1}'\n logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults | ++ grep watcher\n logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=63195c8d1c084032a109516d802e4c7d\n logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | + '[' 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d ']'\n logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n \ logger.go:42: 19:09:21 | watcher-api-scaling/1-deploy-with-defaults | + '[' '' == '' ']'\n logger.go:42: 19:09:21 | watcher-api-scaling/1-deploy-with-defaults | + exit 0\n logger.go:42: 19:09:22 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n \ counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n \ ]\n logger.go:42: 19:09:22 | watcher-api-scaling/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:09:22 | watcher-api-scaling/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults | ++ grep watcher\n logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults | ++ awk '{print $1}'\n logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=63195c8d1c084032a109516d802e4c7d\n logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + '[' 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d ']'\n logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n \ logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + '[' '' == '' ']'\n logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + exit 0\n logger.go:42: 19:09:27 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n \ counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n \ ]\n logger.go:42: 19:09:27 | watcher-api-scaling/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:09:27 | watcher-api-scaling/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults | ++ awk '{print $1}'\n logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults | ++ grep watcher\n logger.go:42: 19:09:31 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=63195c8d1c084032a109516d802e4c7d\n logger.go:42: 19:09:31 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + '[' 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d ']'\n logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n \ logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + '[' '' == '' ']'\n logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + exit 0\n logger.go:42: 19:09:33 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container locally, skip following test\n if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n \ counter=0\n for i in ${env_variables}; do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n \ ]\n logger.go:42: 19:09:33 | watcher-api-scaling/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:09:33 | watcher-api-scaling/1-deploy-with-defaults | ++ grep -c '^watcher'\n logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults | ++ grep watcher\n logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults | ++ awk '{print $1}'\n logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=63195c8d1c084032a109516d802e4c7d\n logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n \ logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | + '[' 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d ']'\n logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q ']'\n logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n \ logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | + '[' '' == '' ']'\n logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | + exit 0\n logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | test step completed 1-deploy-with-defaults\n logger.go:42: 19:09:38 | watcher-api-scaling/2-scale-up-watcher-api | starting test step 2-scale-up-watcher-api\n logger.go:42: 19:09:38 | watcher-api-scaling/2-scale-up-watcher-api | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":3}]'\n \ ]\n logger.go:42: 19:09:38 | watcher-api-scaling/2-scale-up-watcher-api | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 19:09:49 | watcher-api-scaling/2-scale-up-watcher-api | test step completed 2-scale-up-watcher-api\n \ logger.go:42: 19:09:49 | watcher-api-scaling/3-scale-down-watcher-api | starting test step 3-scale-down-watcher-api\n logger.go:42: 19:09:49 | watcher-api-scaling/3-scale-down-watcher-api | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":1}]'\n \ ]\n logger.go:42: 19:09:49 | watcher-api-scaling/3-scale-down-watcher-api | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 19:09:52 | watcher-api-scaling/3-scale-down-watcher-api | test step completed 3-scale-down-watcher-api\n \ logger.go:42: 19:09:52 | watcher-api-scaling/4-scale-down-zero-watcher-api | starting test step 4-scale-down-zero-watcher-api\n logger.go:42: 19:09:52 | watcher-api-scaling/4-scale-down-zero-watcher-api | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":0}]'\n \ ]\n logger.go:42: 19:09:52 | watcher-api-scaling/4-scale-down-zero-watcher-api | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 19:09:53 | watcher-api-scaling/4-scale-down-zero-watcher-api | test step completed 4-scale-down-zero-watcher-api\n \ logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | starting test step 5-cleanup-watcher\n logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c '^watcher'\n logger.go:42: 19:09:56 | watcher-api-scaling/5-cleanup-watcher | + '[' 1 == 0 ']'\n logger.go:42: 19:09:57 | watcher-api-scaling/5-cleanup-watcher | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 19:09:57 | watcher-api-scaling/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:09:57 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c '^watcher'\n logger.go:42: 19:09:59 | watcher-api-scaling/5-cleanup-watcher | + '[' 0 == 0 ']'\n logger.go:42: 19:10:00 | watcher-api-scaling/5-cleanup-watcher | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 19:10:00 | watcher-api-scaling/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type\n logger.go:42: 19:10:00 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c '^watcher'\n logger.go:42: 19:10:02 | watcher-api-scaling/5-cleanup-watcher | + '[' 0 == 0 ']'\n logger.go:42: 19:10:02 | watcher-api-scaling/5-cleanup-watcher | test step completed 5-cleanup-watcher\n logger.go:42: 19:10:02 | watcher-api-scaling | skipping kubernetes event logging\n=== CONT kuttl/harness/deps\n logger.go:42: 19:10:02 | deps | Ignoring infra.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n \ logger.go:42: 19:10:02 | deps | Ignoring keystone.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n logger.go:42: 19:10:02 | deps | Ignoring kustomization.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n \ logger.go:42: 19:10:02 | deps | Ignoring namespace.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n logger.go:42: 19:10:02 | deps | Ignoring telemetry.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n \ logger.go:42: 19:10:02 | deps | Skipping creation of user-supplied namespace: watcher-kuttl-default\n logger.go:42: 19:10:02 | deps | skipping kubernetes event logging\n=== CONT kuttl/harness/common\n logger.go:42: 19:10:02 | common | Ignoring cleanup-assert.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n \ logger.go:42: 19:10:02 | common | Ignoring cleanup-errors.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n logger.go:42: 19:10:02 | common | Ignoring cleanup-watcher.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n logger.go:42: 19:10:02 | common | Ignoring deploy-with-defaults.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n \ logger.go:42: 19:10:02 | common | Skipping creation of user-supplied namespace: watcher-kuttl-default\n logger.go:42: 19:10:02 | common | skipping kubernetes event logging\n=== NAME kuttl\n harness.go:406: run tests finished\n harness.go:514: cleaning up\n harness.go:571: removing temp folder: \"\"\n--- FAIL: kuttl (1502.65s)\n \ --- FAIL: kuttl/harness (0.00s)\n --- FAIL: kuttl/harness/appcred-tests (959.40s)\n --- PASS: kuttl/harness/watcher-notification (36.28s)\n --- PASS: kuttl/harness/watcher-topology (33.59s)\n --- PASS: kuttl/harness/watcher-tls-certs-change (37.89s)\n --- PASS: kuttl/harness/watcher-tls (120.05s)\n --- PASS: kuttl/harness/watcher-rmquser (34.79s)\n --- PASS: kuttl/harness/watcher (114.14s)\n --- PASS: kuttl/harness/watcher-cinder (112.06s)\n --- PASS: kuttl/harness/watcher-api-scaling (54.41s)\n --- PASS: kuttl/harness/deps (0.00s)\n --- PASS: kuttl/harness/common (0.00s)\nFAIL" stdout_lines: - oc kuttl test --v 1 --start-kind=false --config test/kuttl/test-suites/default/config.yaml - === RUN kuttl - ' harness.go:463: starting setup' - ' harness.go:255: running tests using configured kubeconfig.' - ' harness.go:278: Successful connection to cluster at: https://api.crc.testing:6443' - ' harness.go:363: running tests' - ' harness.go:75: going to run test suite with timeout of 300 seconds for each step' - ' harness.go:375: testsuite: test/kuttl/test-suites/default/ has 11 tests' - === RUN kuttl/harness - === RUN kuttl/harness/appcred-tests - === PAUSE kuttl/harness/appcred-tests - === RUN kuttl/harness/common - === PAUSE kuttl/harness/common - === RUN kuttl/harness/deps - === PAUSE kuttl/harness/deps - === RUN kuttl/harness/watcher - === PAUSE kuttl/harness/watcher - === RUN kuttl/harness/watcher-api-scaling - === PAUSE kuttl/harness/watcher-api-scaling - === RUN kuttl/harness/watcher-cinder - === PAUSE kuttl/harness/watcher-cinder - === RUN kuttl/harness/watcher-notification - === PAUSE kuttl/harness/watcher-notification - === RUN kuttl/harness/watcher-rmquser - === PAUSE kuttl/harness/watcher-rmquser - === RUN kuttl/harness/watcher-tls - === PAUSE kuttl/harness/watcher-tls - === RUN kuttl/harness/watcher-tls-certs-change - === PAUSE kuttl/harness/watcher-tls-certs-change - === RUN kuttl/harness/watcher-topology - === PAUSE kuttl/harness/watcher-topology - === CONT kuttl/harness/appcred-tests - ' logger.go:42: 18:45:00 | appcred-tests | Skipping creation of user-supplied namespace: watcher-kuttl-default' - ' logger.go:42: 18:45:00 | appcred-tests/0-cleanup | starting test step 0-cleanup' - ' logger.go:42: 18:45:00 | appcred-tests/0-cleanup | test step completed 0-cleanup' - ' logger.go:42: 18:45:00 | appcred-tests/1-deploy | starting test step 1-deploy' - ' logger.go:42: 18:45:00 | appcred-tests/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl created' - ' logger.go:42: 18:45:00 | appcred-tests/1-deploy | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' oc wait -n "${NS}" watcher watcher-kuttl --for=condition=Ready --timeout=600s]' - ' logger.go:42: 18:45:57 | appcred-tests/1-deploy | watcher.watcher.openstack.org/watcher-kuttl condition met' - ' logger.go:42: 18:45:57 | appcred-tests/1-deploy | test step completed 1-deploy' - ' logger.go:42: 18:45:57 | appcred-tests/2-deploy-appcred | starting test step 2-deploy-appcred' - ' logger.go:42: 18:45:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' old_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' old_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' old_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' oc create configmap appcred-watcher-pre \' - ' --from-literal=api_uid="${old_api_uid}" \' - ' --from-literal=applier_uid="${old_applier_uid}" \' - ' --from-literal=decision_engine_uid="${old_de_uid}" \' - ' --dry-run=client -o yaml | oc apply -n "${NS}" -f -' - ' ' - ' cat <<''EOF'' | oc apply -n "${NS}" -f -' - ' apiVersion: keystone.openstack.org/v1beta1' - ' kind: KeystoneApplicationCredential' - ' metadata:' - ' name: ac-watcher' - ' spec:' - ' secret: os**********et' - ' passwordSelector: WatcherPassword' - ' userName: watcher' - ' roles:' - ' - admin' - ' - service' - ' unrestricted: false' - ' EOF' - ' ' - ' oc patch watcher watcher-kuttl -n "${NS}" --type=merge -p ''{"spec":{"auth":{"applicationCredentialSecret":"ac-watcher-secret"}}}'']' - ' logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | configmap/appcred-watcher-pre created' - ' logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | keystoneapplicationcredential.keystone.openstack.org/ac-watcher created' - ' logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | watcher.watcher.openstack.org/watcher-kuttl patched' - ' logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:45:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:50:58 | appcred-tests/2-deploy-appcred | error: timed out waiting for the condition on keystoneapplicationcredentials/ac-watcher' - ' logger.go:42: 18:50:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:50:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:50:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:29 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:36 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:44 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:51:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:51:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:51:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:29 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:52:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:52:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:52:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:08 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:15 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:44 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:53:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:53:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:53:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:08 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:15 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:36 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:54:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:54:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:54:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:08 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:55:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:55:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:55:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:08 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:15 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:36 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:56:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:56:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:56:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:08 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:15 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:36 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:44 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:57:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:57:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:57:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:15 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:29 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:36 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:44 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:55 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:58:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:58:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:58:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:01 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:08 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:15 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:23 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:29 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:36 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:43 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:44 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:49 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 18:59:59 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 18:59:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 18:59:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:00 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:02 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:03 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:04 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:05 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:06 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:07 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:09 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:10 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:11 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:12 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:13 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:14 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:16 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:17 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:18 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:19 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:20 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:21 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:22 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:24 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:25 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:26 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:27 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:28 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:30 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:31 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:32 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:33 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:34 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:35 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:37 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:38 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:39 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:40 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:41 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:42 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:44 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:45 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:46 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:47 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:48 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:50 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:51 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:52 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:53 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:54 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:56 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:57 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:58 | appcred-tests/2-deploy-appcred | running command: [sh -c set -euo pipefail' - ' NS="${NAMESPACE}"' - ' ' - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' - ' ' - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' - ' if [ -z "${ac_id}" ]; then' - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' - ' exit 1' - ' fi' - ' echo "ac-watcher.status.acID=${ac_id}"' - ' ' - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' | base64 -d)' - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' - ' exit 1' - ' fi' - ' echo "✓ Secret AC_ID matches appcred.status.acID"' - ' ' - ' echo "Waiting for watcher rollouts..."' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine --timeout=300s' - ' ' - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' - ' ' - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' - ' ' - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' - ' echo "ERROR: expected watcher pods to restart after appcred secret became available"' - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' - ' exit 1' - ' fi' - ' echo "✓ watcher pods restarted after appcred secret became available"' - ' ' - ' echo "Checking watcher config contains application_credential_id..."' - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine -- \' - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' - ' echo "✓ watcher config contains expected application_credential_id"]' - ' logger.go:42: 19:00:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...' - ' logger.go:42: 19:00:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not found' - ' logger.go:42: 19:00:59 | appcred-tests/2-deploy-appcred | test step failed 2-deploy-appcred' - ' case.go:396: failed in step 2-deploy-appcred' - ' case.go:398: command "NS=\"${NAMESPACE}\"\\n echo \"Waiting for KeystoneApplicationCredential..." failed, exit status 1' - ' logger.go:42: 19:00:59 | appcred-tests | skipping kubernetes event logging' - === CONT kuttl/harness/watcher-notification - ' logger.go:42: 19:00:59 | watcher-notification | Skipping creation of user-supplied namespace: watcher-kuttl-default' - ' logger.go:42: 19:00:59 | watcher-notification/0-cleanup-watcher | starting test step 0-cleanup-watcher' - ' logger.go:42: 19:00:59 | watcher-notification/0-cleanup-watcher | test step completed 0-cleanup-watcher' - ' logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification | starting test step 1-deploy-with-notification' - ' logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification | Watcher:watcher-kuttl-default/watcher-kuttl created' - ' logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:00:59 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | Now using project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | error: Internal error occurred: error executing command in container: container is not created or running' - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | ++ echo' - ' logger.go:42: 19:01:00 | watcher-notification/1-deploy-with-notification | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | Error from server (NotFound): pods "watcher-kuttl-api-0" not found' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | ++ echo' - ' logger.go:42: 19:01:02 | watcher-notification/1-deploy-with-notification | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:03 | watcher-notification/1-deploy-with-notification | + APIPOD=' - ' logger.go:42: 19:01:04 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:04 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:05 | watcher-notification/1-deploy-with-notification | + APIPOD=' - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:06 | watcher-notification/1-deploy-with-notification | + APIPOD=' - ' logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:07 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:08 | watcher-notification/1-deploy-with-notification | + APIPOD=' - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:09 | watcher-notification/1-deploy-with-notification | + APIPOD=' - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:10 | watcher-notification/1-deploy-with-notification | + APIPOD=' - ' logger.go:42: 19:01:11 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:11 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:12 | watcher-notification/1-deploy-with-notification | + APIPOD=' - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:13 | watcher-notification/1-deploy-with-notification | + APIPOD=' - ' logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:14 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:15 | watcher-notification/1-deploy-with-notification | + APIPOD=' - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:16 | watcher-notification/1-deploy-with-notification | + APIPOD=' - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' - ' logger.go:42: 19:01:17 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:01:18 | watcher-notification/1-deploy-with-notification | error: unable to upgrade connection: container not found ("watcher-api")' - ' logger.go:42: 19:01:18 | watcher-notification/1-deploy-with-notification | ++ echo' - ' logger.go:42: 19:01:18 | watcher-notification/1-deploy-with-notification | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = messagingv2 transport_url = ''rabbit://**********=1'' ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:01:19 | watcher-notification/1-deploy-with-notification | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:20 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' - ' logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = messagingv2 transport_url = ''rabbit://**********=1'' ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:01:21 | watcher-notification/1-deploy-with-notification | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = messagingv2 transport_url = ''rabbit://**********=1'' ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:01:22 | watcher-notification/1-deploy-with-notification | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = messagingv2 transport_url = ''rabbit://**********=1'' ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:01:24 | watcher-notification/1-deploy-with-notification | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = messagingv2 transport_url = ''rabbit://**********=1'' ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:01:26 | watcher-notification/1-deploy-with-notification | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:27 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' - ' logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = messagingv2 transport_url = ''rabbit://**********=1'' ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:01:28 | watcher-notification/1-deploy-with-notification | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ head -1' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ grep -v ''^$''' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_b4b0:c5d9bf6f2eb20eef478aa4903b2c5d4c@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = messagingv2 transport_url = ''rabbit://**********=1'' ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:01:29 | watcher-notification/1-deploy-with-notification | test step completed 1-deploy-with-notification' - ' logger.go:42: 19:01:29 | watcher-notification/2-cleanup-watcher | starting test step 2-cleanup-watcher' - ' logger.go:42: 19:01:35 | watcher-notification/2-cleanup-watcher | test step completed 2-cleanup-watcher' - ' logger.go:42: 19:01:35 | watcher-notification | skipping kubernetes event logging' - === CONT kuttl/harness/watcher-topology - ' logger.go:42: 19:01:35 | watcher-topology | Skipping creation of user-supplied namespace: watcher-kuttl-default' - ' logger.go:42: 19:01:35 | watcher-topology/0-cleanup-watcher | starting test step 0-cleanup-watcher' - ' logger.go:42: 19:01:35 | watcher-topology/0-cleanup-watcher | test step completed 0-cleanup-watcher' - ' logger.go:42: 19:01:35 | watcher-topology/1-deploy-with-topology | starting test step 1-deploy-with-topology' - ' logger.go:42: 19:01:35 | watcher-topology/1-deploy-with-topology | Topology:watcher-kuttl-default/watcher-api created' - ' logger.go:42: 19:01:35 | watcher-topology/1-deploy-with-topology | Watcher:watcher-kuttl-default/watcher-kuttl created' - ' logger.go:42: 19:01:59 | watcher-topology/1-deploy-with-topology | test step completed 1-deploy-with-topology' - ' logger.go:42: 19:01:59 | watcher-topology/2-cleanup-watcher | starting test step 2-cleanup-watcher' - ' logger.go:42: 19:02:00 | watcher-topology/2-cleanup-watcher | running command: [sh -c set -ex' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' - ' ]' - ' logger.go:42: 19:02:00 | watcher-topology/2-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:02:00 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher''' - ' logger.go:42: 19:02:02 | watcher-topology/2-cleanup-watcher | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:02:04 | watcher-topology/2-cleanup-watcher | running command: [sh -c set -ex' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' - ' ]' - ' logger.go:42: 19:02:04 | watcher-topology/2-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:02:04 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher''' - ' logger.go:42: 19:02:06 | watcher-topology/2-cleanup-watcher | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:02:07 | watcher-topology/2-cleanup-watcher | running command: [sh -c set -ex' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' - ' ]' - ' logger.go:42: 19:02:07 | watcher-topology/2-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:02:07 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher''' - ' logger.go:42: 19:02:09 | watcher-topology/2-cleanup-watcher | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:02:09 | watcher-topology/2-cleanup-watcher | test step completed 2-cleanup-watcher' - ' logger.go:42: 19:02:09 | watcher-topology | skipping kubernetes event logging' - === CONT kuttl/harness/watcher-tls-certs-change - ' logger.go:42: 19:02:09 | watcher-tls-certs-change | Skipping creation of user-supplied namespace: watcher-kuttl-default' - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/0-cleanup-watcher | starting test step 0-cleanup-watcher' - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/0-cleanup-watcher | test step completed 0-cleanup-watcher' - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | starting test step 1-deploy-with-tlse' - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-internal-svc created' - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc created' - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl created' - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods "watcher-kuttl-api-0" not found' - ' logger.go:42: 19:02:09 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' - ' logger.go:42: 19:02:10 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:10 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:11 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods "watcher-kuttl-api-0" not found' - ' logger.go:42: 19:02:11 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' - ' logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods "watcher-kuttl-api-0" not found' - ' logger.go:42: 19:02:12 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' - ' logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods "watcher-kuttl-api-0" not found' - ' logger.go:42: 19:02:13 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' - ' logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods "watcher-kuttl-api-0" not found' - ' logger.go:42: 19:02:14 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' - ' logger.go:42: 19:02:15 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:15 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:16 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods "watcher-kuttl-api-0" not found' - ' logger.go:42: 19:02:16 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' - ' logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods "watcher-kuttl-api-0" not found' - ' logger.go:42: 19:02:17 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' - ' logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods "watcher-kuttl-api-0" not found' - ' logger.go:42: 19:02:18 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' - ' logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods "watcher-kuttl-api-0" not found' - ' logger.go:42: 19:02:19 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' - ' logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | Error from server (NotFound): pods "watcher-kuttl-api-0" not found' - ' logger.go:42: 19:02:20 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' - ' logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | error: unable to upgrade connection: container not found ("watcher-api")' - ' logger.go:42: 19:02:22 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' - ' logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | error: unable to upgrade connection: container not found ("watcher-api")' - ' logger.go:42: 19:02:23 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' - ' logger.go:42: 19:02:24 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:24 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:25 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:26 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:27 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:29 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:30 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:31 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:32 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:33 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:33 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:34 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ' - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:35 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 --decode' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/1-deploy-with-tlse | test step completed 1-deploy-with-tlse' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | starting test step 2-change-public-svc-certificate' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-public-svc --type=''json'' -p=''[{"op": "replace", "path": "/spec/dnsNames", "value":[''watcher-public.watcher-kuttl-default.svc'', ''watcher-public.watcher-kuttl-default.svc.cluster.local'']}]''' - ' ]' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | certificate.cert-manager.io/watcher-public-svc patched' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh -c set -euxo pipefail' - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | + svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ base64 --decode' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | + secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MIIDszCCAhugAwIBAgIRAJRTcdJPmajlfohlpn9xjWIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMDlaFw0zMTAz' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMDlaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLrpcC' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | WI9OS9u34c32rlOb4xHqmZ5mGDFoSDhWXKLu+c34qr5uZt0BECOGMpOavTw1VGcZ' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | jx8EnDRa+M/GKSAC9pO3u9V3KbPtjsHOT3R6//Ryfe/PqllwNJID7xO3ZVP1s+gP' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Jdd2KNTH2+U0K2Z0aVvEzecAnIL9xz83CJcnDd5ucZWuMsvGrS8WhS6leVBbBZ3Y' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | PNTvqt1kN6e41atpZxkjy8CwlpvI1tGIaOLrR0feZ4FsSFm6+gBgi+fHc+HAtmQm' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | zbQ+vkdtfRGRvuIavvM9CPhpUUXHFqp8QbgHFBewdHpKji6soXOk/z+paX1xQsvR' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | tZBmu2oBaSxLZy/jAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAhU0QTdd5DhFG' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | zipxYrlc2zqoe3acyW3U8bz/j2+MhMpLCq0PIQdA03Exi4pHhSkKQf7OXzeqRofu' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | w+aIt3xQEAO7L0rx8fPbuDSeHxdA2Ars9/jVZHkYS/CnjJNoBwme66+IMOFgQQTZ' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rtkDoGE5BzVSavS67dJYudZL4FHSIhg2iHvO/gGL012fDaf2FqfWezpHbTRP/tUj' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | x/m/rwjGBltCvd1uurr3Ur4v8PzF9RrZZemvFgzk6iNMV16B/A9Uf++OC+a3TbKX' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GQUhKTUmDPqbCfaseiUXI5dbgn4J2Rei9Y1J/+ri0tYvQVeAQ35LYmH14h8FQ9Gq' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rrhJoCwNCigrHCugmU5GBt0Epfoo9V9wqQfj4AuPI1WwG1PtL+2R38Y02o6hJaTe' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | LLLilFK+JtOtaJzehN8Mvwphr6HNrIXXzFqgbqidWCVC0Us0Nfk0pZ3fuVBxXUer' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | av0z/6DW2yzvXV2eYXemSVl9aX7yXwnnWlsFzCbsyZAVZYF1z/og' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:36 | watcher-tls-certs-change/2-change-public-svc-certificate | + exit 1' - ' logger.go:42: 19:02:37 | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh -c set -euxo pipefail' - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:37 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:38 | watcher-tls-certs-change/2-change-public-svc-certificate | error: unable to upgrade connection: container not found ("watcher-api")' - ' logger.go:42: 19:02:38 | watcher-tls-certs-change/2-change-public-svc-certificate | + svc_cert=' - ' logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh -c set -euxo pipefail' - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate | error: unable to upgrade connection: container not found ("watcher-api")' - ' logger.go:42: 19:02:39 | watcher-tls-certs-change/2-change-public-svc-certificate | + svc_cert=' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh -c set -euxo pipefail' - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)' - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | + svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ base64 --decode' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | + secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6zCCAlOgAwIBAgIRAJJH32cSXoB1MYirv9r0NvIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMDkxOTAyMzZaFw0zMTAz' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MDgxOTAyMzZaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFW5Op' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | r1zi2ekF4d4c77G/B0BYKhRYcp+kv1f3ZENJc5rxjHnB3fijRwb4Jl8wjlyWROro' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | +JoqcTVI8IxAoBHpoT5415oFZVtKX2xPoT6WdhS1JZWo4haVqLrtztIxso7JEoxP' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | h236WtruCvRcWDdQzlveMu8QKJLoZtcUBE8WfSPFnAW3ab6V8zE9vvWqaFKOygY8' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | xuta0nzlIwwH/qBANL0RM4CYO5ZtbC+gk6BES/PHoJtjg/4c5oQKj0Tpon8U/ku3' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | gg62l+LfGdGLAJwp0Bncf3LbDZybifxj2uqMaYMjN9ftKr7r2a+k7QtlnNO5uNdr' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | B9lHcmEPtN51VO8LAgMBAAGjgccwgcQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUFoTEDV3EOE0z' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | rYm2tWepPVohIt0wbgYDVR0RAQH/BGQwYoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2Y4I2d2F0Y2hlci1wdWJsaWMud2F0Y2hlci1rdXR0' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | bC1kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3DQEBDAUAA4IBgQAS' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Z8LUmouBW7FXush1A21UWxyRFvHSnqPgkGcZA4cQHZGiEYiYimjC7A0dPPaomGXr' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | XXfyTVItPWXxnco1hGvBugP5pmeq5Xh8wIqF8sl392Qh6ZQzOjyHISOQJpBS88W1' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eZc2X4vMvO95CFURXHyCzoWv6A1E84FIDxniRvIdFgB29EdFlnus4ilu6qR3Itvj' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | Ok9krsqGjCEBXGxtShdTvIV6bGWP/GPUwU1/42wuHNymjxYlb7oVBJaCyrYAAMDo' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | 8nMK0dCOQSZNBEkkhcROmqya4iimT80gDF0Qvr071UHiIkIUQmHw46C2nACuDv4p' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | KOmpO/vF0h80U9JY2f0xt+RkMWkHyONNE9VT2llolGTwlQxxO4ls8L8fNdn5eGxq' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | eKDCsBJVVsWFX9Tvup1W87tS2KyYqCzUNv+8OhYEgxnUJNEphnenRqtbLZjXC1Xu' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | tpD/FKNloCLl2opmZHSso4PNo3osoq8D3d4GXT30pcDfw5ncqBf9QFgWsglrAsg=' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/2-change-public-svc-certificate | test step completed 2-change-public-svc-certificate' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/3-change-internal-svc-certificate | starting test step 3-change-internal-svc-certificate' - ' logger.go:42: 19:02:40 | watcher-tls-certs-change/3-change-internal-svc-certificate | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-internal-svc --type=''json'' -p=''[{"op": "replace", "path": "/spec/dnsNames", "value":[''watcher-internal.watcher-kuttl-default.svc'', ''watcher-internal.watcher-kuttl-default.svc.cluster.local'']}]''' - ' ]' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | certificate.cert-manager.io/watcher-internal-svc patched' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | running command: [sh -c set -euxo pipefail' - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)' - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' - ' # ensure that the svc secret and cert secret match' - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | + svc_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | ++ base64 --decode' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | + secret_cert=''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | -----END CERTIFICATE-----''' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | + ''['' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MIIDtzCCAh+gAwIBAgIRAMn7mox9hpHbY1hgXU4vudIwDQYJKoZIhvcNAQEMBQAw' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMwOTE5MDIwOVoXDTMx' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | MDMwODE5MDIwOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMC4' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | F2Om9chmdDb7fNuW/6MZoBmdIQs1AYONHWueamy81VSeYimuuMzyzAnAMPDyBv4q' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | PIJGFFvB3CvY3UWYf+eCSN6xWwTkyP5dHy+JpLf97jE+GfN8FgeHKfuXA4IrhGMD' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 1iq5xH6VZFi/cutr7rStiwBgTsk2b+DQxJt4EzMGKmsQzo5XasZ5Ammml96DKf96' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | I27f4WUvyLprsaLSlO5hdNS8FzVPDvMvJiAqsAcDcjAWvJ6cdBC9Xu3+t4lL1js+' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 3iV9VAVN3aS4XVc60Y/duhZvLhiXMRk+c5V5RhBU1z/3KmxQDG4SsJUjkItkyffg' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Q886Z2vZRPtfKCBByuECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSPUvO+tMQV' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | YvbPrhbDY/G6fS6eOzA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBADmCLbke' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | M/o4MbpaQGspd81XGEy5XcrBARQEA0T8YeHio3Dvv2zAckjvQjuXDycyVttu8Eee' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | /kJuwNDdhYOwW7yj3kpzirz246/I/SyWl2zlClA2JRkZcxKavdtjLZgfE00IryIp' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | Ktuyv+w1FGZumbP932hJo/BuWD7pb7iPx/xqMyl5/Tm/sT71ND/tQP0pFZeERh/N' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 9jBthBbr5rwzNDlmp2Ur2J67zYcJfFbbOkp9cXW9jiA5LnTYzrwDEZC3YIpsk6wb' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | NgecufricPtJt56x2UbTB+YVhLQOI36KYu3qIVyR0qMS+0f79TFVguiwBObIEPJz' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | 8WobsjK8dB4PeHv6yAFnV2NiQVwwYXHf8NltMeC6Z0qpupQX+oAE3gC7wQlqS6nH' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | bWzpSxzm4Rjd+BR3Df+RjPNnsWy92EyRWkTppHjr3+viI9fl5DPmQvgrT3yu8PR+' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | fZ4maot9J/11INLPiUSXzeXl09NkS+FpexkAfjuS7rSR6uZ1gLHKGPpN3w==' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | -----END CERTIFICATE-----'' '']''' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/3-change-internal-svc-certificate | test step completed 3-change-internal-svc-certificate' - ' logger.go:42: 19:02:41 | watcher-tls-certs-change/4-cleanup-watcher | starting test step 4-cleanup-watcher' - ' logger.go:42: 19:02:47 | watcher-tls-certs-change/4-cleanup-watcher | test step completed 4-cleanup-watcher' - ' logger.go:42: 19:02:47 | watcher-tls-certs-change/5-clenaup-certs | starting test step 5-clenaup-certs' - ' logger.go:42: 19:02:47 | watcher-tls-certs-change/5-clenaup-certs | test step completed 5-clenaup-certs' - ' logger.go:42: 19:02:47 | watcher-tls-certs-change | skipping kubernetes event logging' - === CONT kuttl/harness/watcher-tls - ' logger.go:42: 19:02:47 | watcher-tls | Skipping creation of user-supplied namespace: watcher-kuttl-default' - ' logger.go:42: 19:02:47 | watcher-tls/0-cleanup-watcher | starting test step 0-cleanup-watcher' - ' logger.go:42: 19:02:47 | watcher-tls/0-cleanup-watcher | test step completed 0-cleanup-watcher' - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | starting test step 1-deploy-with-tlse' - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-internal-svc created' - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc created' - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl created' - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' # check that both endpoints have https set' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 2 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:02:47 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' - ' logger.go:42: 19:02:49 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:02:50 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' # check that both endpoints have https set' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 2 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:02:50 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:02:50 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' - ' logger.go:42: 19:02:53 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:02:54 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' # check that both endpoints have https set' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 2 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:02:54 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' - ' logger.go:42: 19:02:54 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | ++ grep watcher' - ' logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:02:56 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}''' - ' logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:02:58 | watcher-tls/1-deploy-with-tlse | + ''['' -n '''' '']''' - ' logger.go:42: 19:03:00 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' # check that both endpoints have https set' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 2 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:03:00 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:03:00 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' - ' logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | ++ grep watcher' - ' logger.go:42: 19:03:02 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}''' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + grep infra-optim' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list' - ' logger.go:42: 19:03:04 | watcher-tls/1-deploy-with-tlse | ++ grep -c https' - ' logger.go:42: 19:03:06 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:03:07 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' # check that both endpoints have https set' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 2 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:03:07 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:03:07 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' - ' logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}''' - ' logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | ++ grep watcher' - ' logger.go:42: 19:03:09 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:03:11 | watcher-tls/1-deploy-with-tlse | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + grep infra-optim' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list' - ' logger.go:42: 19:03:12 | watcher-tls/1-deploy-with-tlse | ++ grep -c https' - ' logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:03:14 | watcher-tls/1-deploy-with-tlse | + exit 0' - ' logger.go:42: 19:03:16 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' # check that both endpoints have https set' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 2 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:03:16 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:03:16 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' - ' logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}''' - ' logger.go:42: 19:03:18 | watcher-tls/1-deploy-with-tlse | ++ grep watcher' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + grep infra-optim' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | ++ grep -c https' - ' logger.go:42: 19:03:20 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list' - ' logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | + exit 0' - ' logger.go:42: 19:03:22 | watcher-tls/1-deploy-with-tlse | test step completed 1-deploy-with-tlse' - ' logger.go:42: 19:03:22 | watcher-tls/2-patch-mtls | starting test step 2-patch-mtls' - ' logger.go:42: 19:03:22 | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail' - ' oc patch oscp -n $NAMESPACE openstack --type=''json'' -p=''[{"op": "replace", "path": "/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode", "value": "Request"}]''' - ' ]' - ' logger.go:42: 19:03:22 | watcher-tls/2-patch-mtls | + oc patch oscp -n watcher-kuttl-default openstack --type=json ''-p=[{"op": "replace", "path": "/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode", "value": "Request"}]''' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | openstackcontrolplane.core.openstack.org/openstack patched' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail' - ' ' - ' oc project ${NAMESPACE}' - ' # Get pod names for each watcher service' - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' - ' ' - ' # Verify memcached mTLS config parameters in watcher-api config' - ' if [ -n "${APIPOD}" ]; then' - ' echo "Checking watcher-api config..."' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' # Verify mTLS config parameters in memcached backend config' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ' - ' # Verify memcached mTLS config parameters in watcher-applier config' - ' if [ -n "${APPLIERPOD}" ]; then' - ' echo "Checking watcher-applier config..."' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ' - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' - ' if [ -n "${DECISIONENGINEPOD}" ]; then' - ' echo "Checking watcher-decision-engine config..."' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api config...''' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | Checking watcher-api config...' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ''' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:23 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail' - ' ' - ' oc project ${NAMESPACE}' - ' # Get pod names for each watcher service' - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' - ' ' - ' # Verify memcached mTLS config parameters in watcher-api config' - ' if [ -n "${APIPOD}" ]; then' - ' echo "Checking watcher-api config..."' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' # Verify mTLS config parameters in memcached backend config' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ' - ' # Verify memcached mTLS config parameters in watcher-applier config' - ' if [ -n "${APPLIERPOD}" ]; then' - ' echo "Checking watcher-applier config..."' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ' - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' - ' if [ -n "${DECISIONENGINEPOD}" ]; then' - ' echo "Checking watcher-decision-engine config..."' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' - ' logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:03:24 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api config...''' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | Checking watcher-api config...' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ''' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | error: cannot exec into a container in a completed pod; current phase is Failed' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | ++ echo' - ' logger.go:42: 19:03:25 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail' - ' ' - ' oc project ${NAMESPACE}' - ' # Get pod names for each watcher service' - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' - ' ' - ' # Verify memcached mTLS config parameters in watcher-api config' - ' if [ -n "${APIPOD}" ]; then' - ' echo "Checking watcher-api config..."' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' # Verify mTLS config parameters in memcached backend config' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ' - ' # Verify memcached mTLS config parameters in watcher-applier config' - ' if [ -n "${APPLIERPOD}" ]; then' - ' echo "Checking watcher-applier config..."' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ' - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' - ' if [ -n "${DECISIONENGINEPOD}" ]; then' - ' echo "Checking watcher-decision-engine config..."' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' - ' logger.go:42: 19:03:26 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api config...''' - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | Checking watcher-api config...' - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ''' - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | error: unable to upgrade connection: container not found ("watcher-api")' - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | ++ echo' - ' logger.go:42: 19:03:27 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail' - ' ' - ' oc project ${NAMESPACE}' - ' # Get pod names for each watcher service' - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' - ' ' - ' # Verify memcached mTLS config parameters in watcher-api config' - ' if [ -n "${APIPOD}" ]; then' - ' echo "Checking watcher-api config..."' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' # Verify mTLS config parameters in memcached backend config' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ' - ' # Verify memcached mTLS config parameters in watcher-applier config' - ' if [ -n "${APPLIERPOD}" ]; then' - ' echo "Checking watcher-applier config..."' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ' - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' - ' if [ -n "${DECISIONENGINEPOD}" ]; then' - ' echo "Checking watcher-decision-engine config..."' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api config...''' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | Checking watcher-api config...' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ''' - ' logger.go:42: 19:03:28 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile = /etc/pki/tls/private/mtls.key''' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt''' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled = true''' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' - ' logger.go:42: 19:03:29 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-applier-0 '']''' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-applier config...''' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | Checking watcher-applier config...' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ''' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile = /etc/pki/tls/private/mtls.key''' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt''' - ' logger.go:42: 19:03:30 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled = true''' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-decision-engine-0 '']''' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-decision-engine config...''' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine config...' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ''' - ' logger.go:42: 19:03:31 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:32 | watcher-tls/2-patch-mtls | error: Internal error occurred: error executing command in container: container is not created or running' - ' logger.go:42: 19:03:32 | watcher-tls/2-patch-mtls | ++ echo' - ' logger.go:42: 19:03:32 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:03:33 | watcher-tls/2-patch-mtls | running command: [sh -c set -euxo pipefail' - ' ' - ' oc project ${NAMESPACE}' - ' # Get pod names for each watcher service' - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' - ' ' - ' # Verify memcached mTLS config parameters in watcher-api config' - ' if [ -n "${APIPOD}" ]; then' - ' echo "Checking watcher-api config..."' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' # Verify mTLS config parameters in memcached backend config' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ' - ' # Verify memcached mTLS config parameters in watcher-applier config' - ' if [ -n "${APPLIERPOD}" ]; then' - ' echo "Checking watcher-applier config..."' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ' - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' - ' if [ -n "${DECISIONENGINEPOD}" ]; then' - ' echo "Checking watcher-decision-engine config..."' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "memcache_tls_enabled = true") == 1 ]' - ' ' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:03:33 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine -o ''jsonpath={.items[0].metadata.name}''' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api config...''' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | Checking watcher-api config...' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ''' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile = /etc/pki/tls/private/mtls.key''' - ' logger.go:42: 19:03:34 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt''' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled = true''' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' - ' logger.go:42: 19:03:35 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-applier-0 '']''' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-applier config...''' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | Checking watcher-applier config...' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ''' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile = /etc/pki/tls/private/mtls.key''' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt''' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled = true''' - ' logger.go:42: 19:03:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-decision-engine-0 '']''' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-decision-engine config...''' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine config...' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ''' - ' logger.go:42: 19:03:37 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile = /etc/pki/tls/private/mtls.key''' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt''' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled = true''' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' - ' logger.go:42: 19:03:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_e081:d3968b1f26a8ccd3b39e9ec83f80c1d3@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:39 | watcher-tls/2-patch-mtls | test step completed 2-patch-mtls' - ' logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | starting test step 3-disable-podlevel-tls' - ' logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": "replace", "path": "/spec/apiServiceTemplate/tls/api", "value":{ "internal": {}, "public": {} }}]''' - ' ]' - ' logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | watcher.watcher.openstack.org/watcher-kuttl patched' - ' logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # check that watcher internal endpoint does not use https' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]' - ' # check that watcher public endpoint does use https' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep public | [ $(grep -c https) == 1 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:03:39 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' - ' logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' - ' logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print $1}''' - ' logger.go:42: 19:03:41 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:03:43 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:03:43 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list' - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + grep internal' - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' - ' logger.go:42: 19:03:44 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' - ' logger.go:42: 19:03:46 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 0 '']''' - ' logger.go:42: 19:03:47 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # check that watcher internal endpoint does not use https' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]' - ' # check that watcher public endpoint does use https' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep public | [ $(grep -c https) == 1 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:03:47 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:03:47 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' - ' logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' - ' logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:03:49 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print $1}''' - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + grep internal' - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list' - ' logger.go:42: 19:03:51 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' - ' logger.go:42: 19:03:53 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 0 '']''' - ' logger.go:42: 19:03:54 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # check that watcher internal endpoint does not use https' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]' - ' # check that watcher public endpoint does use https' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep public | [ $(grep -c https) == 1 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:03:54 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:03:54 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' - ' logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' - ' logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print $1}''' - ' logger.go:42: 19:03:56 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list' - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + grep internal' - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' - ' logger.go:42: 19:03:58 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' - ' logger.go:42: 19:04:00 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 0 '']''' - ' logger.go:42: 19:04:01 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # check that watcher internal endpoint does not use https' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]' - ' # check that watcher public endpoint does use https' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep public | [ $(grep -c https) == 1 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:04:01 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:04:01 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' - ' logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print $1}''' - ' logger.go:42: 19:04:04 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + grep internal' - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' - ' logger.go:42: 19:04:06 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list' - ' logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | + grep public' - ' logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' - ' logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' - ' logger.go:42: 19:04:08 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list' - ' logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:04:10 | watcher-tls/3-disable-podlevel-tls | + exit 0' - ' logger.go:42: 19:04:11 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # check that watcher internal endpoint does not use https' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]' - ' # check that watcher public endpoint does use https' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep public | [ $(grep -c https) == 1 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:04:11 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:04:11 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' - ' logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' - ' logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print $1}''' - ' logger.go:42: 19:04:13 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + grep internal' - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list' - ' logger.go:42: 19:04:16 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' - ' logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | + grep public' - ' logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' - ' logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' - ' logger.go:42: 19:04:18 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list' - ' logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | + exit 0' - ' logger.go:42: 19:04:20 | watcher-tls/3-disable-podlevel-tls | test step completed 3-disable-podlevel-tls' - ' logger.go:42: 19:04:20 | watcher-tls/4-deploy-without-route | starting test step 4-deploy-without-route' - ' logger.go:42: 19:04:20 | watcher-tls/4-deploy-without-route | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": "replace", "path": "/spec/apiServiceTemplate/override", "value":{"service": { "internal": {}, "public": { "metadata": { "annotations": { "metallb.universe.tf/address-pool": "ctlplane", "metallb.universe.tf/allow-shared-ip": "ctlplane" } }, "spec": { "type": "LoadBalancer" } } } }}]''' - ' ]' - ' logger.go:42: 19:04:20 | watcher-tls/4-deploy-without-route | watcher.watcher.openstack.org/watcher-kuttl patched' - ' logger.go:42: 19:04:21 | watcher-tls/4-deploy-without-route | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:04:21 | watcher-tls/4-deploy-without-route | ++ grep -c ''^watcher''' - ' logger.go:42: 19:04:21 | watcher-tls/4-deploy-without-route | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route | ++ awk ''{print $1}''' - ' logger.go:42: 19:04:23 | watcher-tls/4-deploy-without-route | ++ grep watcher' - ' logger.go:42: 19:04:24 | watcher-tls/4-deploy-without-route | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:04:24 | watcher-tls/4-deploy-without-route | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:04:25 | watcher-tls/4-deploy-without-route | + exit 0' - ' logger.go:42: 19:04:26 | watcher-tls/4-deploy-without-route | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:04:26 | watcher-tls/4-deploy-without-route | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:04:26 | watcher-tls/4-deploy-without-route | ++ grep -c ''^watcher''' - ' logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route | ++ awk ''{print $1}''' - ' logger.go:42: 19:04:28 | watcher-tls/4-deploy-without-route | ++ grep watcher' - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | + exit 0' - ' logger.go:42: 19:04:30 | watcher-tls/4-deploy-without-route | test step completed 4-deploy-without-route' - ' logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | starting test step 5-disable-tls' - ' logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": "replace", "path": "/spec/apiServiceTemplate/override", "value":{}}]''' - ' ]' - ' logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | watcher.watcher.openstack.org/watcher-kuttl patched' - ' logger.go:42: 19:04:30 | watcher-tls/5-disable-tls | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' # check that no watcher endpoint uses https' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 0 ]' - ' ]' - ' logger.go:42: 19:04:31 | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:04:31 | watcher-tls/5-disable-tls | ++ grep -c ''^watcher''' - ' logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | ++ grep watcher' - ' logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | ++ awk ''{print $1}''' - ' logger.go:42: 19:04:33 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:04:35 | watcher-tls/5-disable-tls | + exit 0' - ' logger.go:42: 19:04:36 | watcher-tls/5-disable-tls | running command: [sh -c set -euxo pipefail' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' # check that no watcher endpoint uses https' - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) == 0 ]' - ' ]' - ' logger.go:42: 19:04:36 | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:04:36 | watcher-tls/5-disable-tls | ++ grep -c ''^watcher''' - ' logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | ++ grep watcher' - ' logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | ++ awk ''{print $1}''' - ' logger.go:42: 19:04:38 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + SERVICEID=1881642dbc2749d28b8c91d8865efef3' - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + ''['' 1881642dbc2749d28b8c91d8865efef3 == 1881642dbc2749d28b8c91d8865efef3 '']''' - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | + exit 0' - ' logger.go:42: 19:04:40 | watcher-tls/5-disable-tls | test step completed 5-disable-tls' - ' logger.go:42: 19:04:40 | watcher-tls/6-cleanup-watcher | starting test step 6-cleanup-watcher' - ' logger.go:42: 19:04:41 | watcher-tls/6-cleanup-watcher | running command: [sh -c set -ex' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' - ' ]' - ' logger.go:42: 19:04:41 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:04:41 | watcher-tls/6-cleanup-watcher | ++ grep -c ''^watcher''' - ' logger.go:42: 19:04:43 | watcher-tls/6-cleanup-watcher | + ''['' 1 == 0 '']''' - ' logger.go:42: 19:04:44 | watcher-tls/6-cleanup-watcher | running command: [sh -c set -ex' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' - ' ]' - ' logger.go:42: 19:04:44 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:04:44 | watcher-tls/6-cleanup-watcher | ++ grep -c ''^watcher''' - ' logger.go:42: 19:04:47 | watcher-tls/6-cleanup-watcher | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:04:47 | watcher-tls/6-cleanup-watcher | test step completed 6-cleanup-watcher' - ' logger.go:42: 19:04:47 | watcher-tls/7-cleanup-certs | starting test step 7-cleanup-certs' - ' logger.go:42: 19:04:47 | watcher-tls/7-cleanup-certs | test step completed 7-cleanup-certs' - ' logger.go:42: 19:04:47 | watcher-tls | skipping kubernetes event logging' - === CONT kuttl/harness/watcher-rmquser - ' logger.go:42: 19:04:47 | watcher-rmquser | Skipping creation of user-supplied namespace: watcher-kuttl-default' - ' logger.go:42: 19:04:47 | watcher-rmquser/0-cleanup-watcher | starting test step 0-cleanup-watcher' - ' logger.go:42: 19:04:47 | watcher-rmquser/0-cleanup-watcher | test step completed 0-cleanup-watcher' - ' logger.go:42: 19:04:47 | watcher-rmquser/1-deploy | starting test step 1-deploy' - ' logger.go:42: 19:04:47 | watcher-rmquser/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl created' - ' logger.go:42: 19:04:47 | watcher-rmquser/1-deploy | running command: [sh -c set -euxo pipefail' - ' ' - ' # Wait for Watcher to be Ready' - ' kubectl wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE --timeout=300s' - ' ' - ' # Verify WatcherNotificationTransportURLReady condition exists and is True' - ' kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath=''{.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}'' | grep -q "True"' - ' echo "WatcherNotificationTransportURLReady condition is True"' - ' ' - ' # Count TransportURL CRs - should be exactly 2 (one for messaging, one for notifications)' - ' transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep "watcher-kuttl-watcher-transport" | wc -l)' - ' notification_transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep "watcher-kuttl-watcher-notification" | wc -l)' - ' ' - ' if [ "$transport_count" -ne "1" ]; then' - ' echo "Expected 1 watcher-transport TransportURL, found $transport_count"' - ' exit 1' - ' fi' - ' ' - ' if [ "$notification_transport_count" -ne "1" ]; then' - ' echo "Expected 1 notification-transport TransportURL, found $notification_transport_count"' - ' exit 1' - ' fi' - ' ' - ' echo "Correctly found 2 TransportURLs (separate clusters: transport and notification)"' - ' ' - ' # Verify watcher-transport has correct user and vhost' - ' transport_user=$(kubectl get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath=''{.spec.username}'')' - ' transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')' - ' if [ "$transport_user" != "watcher-rpc" ]; then' - ' echo "Expected watcher-transport username ''watcher-rpc'', found ''$transport_user''"' - ' exit 1' - ' fi' - ' if [ "$transport_vhost" != "watcher-rpc" ]; then' - ' echo "Expected watcher-transport vhost ''watcher-rpc'', found ''$transport_vhost''"' - ' exit 1' - ' fi' - ' echo "Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)"' - ' ' - ' # Verify notification-transport has correct user and vhost' - ' notif_user=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath=''{.spec.username}'')' - ' notif_vhost=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')' - ' if [ "$notif_user" != "watcher-notifications" ]; then' - ' echo "Expected notification-transport username ''watcher-notifications'', found ''$notif_user''"' - ' exit 1' - ' fi' - ' if [ "$notif_vhost" != "watcher-notifications" ]; then' - ' echo "Expected notification-transport vhost ''watcher-notifications'', found ''$notif_vhost''"' - ' exit 1' - ' fi' - ' echo "Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)"' - ' ' - ' # Verify that watcher.conf contains the notifications transport_url' - ' WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l "service=watcher-api" -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)' - ' if [ -z "${WATCHER_API_POD}" ]; then' - ' echo "No watcher-api pod found"' - ' exit 1' - ' fi' - ' # Verify RPC transport_url in DEFAULT section' - ' rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E ''^\[DEFAULT\]'' -A 50 | grep ''transport_url'' | head -1 || true)' - ' if [ -z "$rpc_transport_url" ]; then' - ' echo "transport_url not found in DEFAULT section"' - ' exit 1' - ' fi' - ' echo "Found RPC transport_url: $rpc_transport_url"' - ' ' - ' # Verify the RPC transport_url contains the correct vhost (watcher-rpc)' - ' if ! echo "$rpc_transport_url" | grep -q ''/watcher-rpc''; then' - ' echo "RPC transport_url does not contain expected vhost ''/watcher-rpc''"' - ' exit 1' - ' fi' - ' echo "Successfully verified vhost ''watcher-rpc'' in RPC transport_url"' - ' ' - ' # Verify the RPC transport_url contains the correct username (watcher-rpc)' - ' if ! echo "$rpc_transport_url" | grep -q ''watcher-rpc:''; then' - ' echo "RPC transport_url does not contain expected username ''watcher-rpc:''"' - ' exit 1' - ' fi' - ' echo "Successfully verified username ''watcher-rpc'' in RPC transport_url"' - ' ' - ' # Verify oslo_messaging_notifications section has transport_url configured' - ' notif_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -A 5 ''\[oslo_messaging_notifications\]'' | grep ''transport_url'' || true)' - ' if [ -z "$notif_transport_url" ]; then' - ' echo "transport_url not found in oslo_messaging_notifications section"' - ' exit 1' - ' fi' - ' echo "Found notifications transport_url: $notif_transport_url"' - ' ' - ' # Verify the notifications transport_url contains the correct vhost (watcher-notifications)' - ' if ! echo "$notif_transport_url" | grep -q ''/watcher-notifications''; then' - ' echo "Notifications transport_url does not contain expected vhost ''/watcher-notifications''"' - ' exit 1' - ' fi' - ' echo "Successfully verified vhost ''watcher-notifications'' in notifications transport_url"' - ' ' - ' # Verify the notifications transport_url contains the correct username (watcher-notifications)' - ' if ! echo "$notif_transport_url" | grep -q ''watcher-notifications:''; then' - ' echo "Notifications transport_url does not contain expected username ''watcher-notifications:''"' - ' exit 1' - ' fi' - ' echo "Successfully verified username ''watcher-notifications'' in notifications transport_url"' - ' ' - ' exit 0' - ' ]' - ' logger.go:42: 19:04:47 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready watcher/watcher-kuttl -n watcher-kuttl-default --timeout=300s' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | watcher.watcher.openstack.org/watcher-kuttl condition met' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl -n watcher-kuttl-default -o ''jsonpath={.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + grep -q True' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo ''WatcherNotificationTransportURLReady condition is True''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady condition is True' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl -n watcher-kuttl-default -o name' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ wc -l' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + transport_count=1' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl -n watcher-kuttl-default -o name' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ wc -l' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + notification_transport_count=1' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo ''Correctly found 2 TransportURLs (separate clusters: transport and notification)''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs (separate clusters: transport and notification)' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.username}''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + transport_user=watcher-rpc' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.vhost}''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + transport_vhost=watcher-rpc' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!='' watcher-rpc '']''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!='' watcher-rpc '']''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo ''Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default -o ''jsonpath={.spec.username}''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + notif_user=watcher-notifications' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default -o ''jsonpath={.spec.vhost}''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + ''['' watcher-notifications ''!='' watcher-notifications '']''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + ''['' watcher-notifications ''!='' watcher-notifications '']''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | + echo ''Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)''' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ kubectl get pods -n watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name --no-headers' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ head -1' - ' logger.go:42: 19:05:12 | watcher-rmquser/1-deploy | ++ grep -v ''^$''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + ''['' -z watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep -E ''^\[DEFAULT\]'' -A 50' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep transport_url' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ head -1' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + rpc_transport_url=''transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url = rabbit://**********=1'' '']''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''Found RPC transport_url: transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Found RPC transport_url: transport_url = rabbit://**********=1' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''Successfully verified vhost ''\''''watcher-rpc''\'''' in RPC transport_url''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Successfully verified vhost ''watcher-rpc'' in RPC transport_url' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''Successfully verified username ''\''''watcher-rpc''\'''' in RPC transport_url''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Successfully verified username ''watcher-rpc'' in RPC transport_url' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep -A 5 ''\[oslo_messaging_notifications\]''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | ++ grep transport_url' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + notif_transport_url=''transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url = rabbit://**********=1'' '']''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''Found notifications transport_url: transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Found notifications transport_url: transport_url = rabbit://**********=1' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''Successfully verified vhost ''\''''watcher-notifications''\'''' in notifications transport_url''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Successfully verified vhost ''watcher-notifications'' in notifications transport_url' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + echo ''Successfully verified username ''\''''watcher-notifications''\'''' in notifications transport_url''' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | Successfully verified username ''watcher-notifications'' in notifications transport_url' - ' logger.go:42: 19:05:13 | watcher-rmquser/1-deploy | + exit 0' - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | running command: [sh -c set -euxo pipefail' - ' ' - ' # Wait for Watcher to be Ready' - ' kubectl wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE --timeout=300s' - ' ' - ' # Verify WatcherNotificationTransportURLReady condition exists and is True' - ' kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath=''{.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}'' | grep -q "True"' - ' echo "WatcherNotificationTransportURLReady condition is True"' - ' ' - ' # Count TransportURL CRs - should be exactly 2 (one for messaging, one for notifications)' - ' transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep "watcher-kuttl-watcher-transport" | wc -l)' - ' notification_transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep "watcher-kuttl-watcher-notification" | wc -l)' - ' ' - ' if [ "$transport_count" -ne "1" ]; then' - ' echo "Expected 1 watcher-transport TransportURL, found $transport_count"' - ' exit 1' - ' fi' - ' ' - ' if [ "$notification_transport_count" -ne "1" ]; then' - ' echo "Expected 1 notification-transport TransportURL, found $notification_transport_count"' - ' exit 1' - ' fi' - ' ' - ' echo "Correctly found 2 TransportURLs (separate clusters: transport and notification)"' - ' ' - ' # Verify watcher-transport has correct user and vhost' - ' transport_user=$(kubectl get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath=''{.spec.username}'')' - ' transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')' - ' if [ "$transport_user" != "watcher-rpc" ]; then' - ' echo "Expected watcher-transport username ''watcher-rpc'', found ''$transport_user''"' - ' exit 1' - ' fi' - ' if [ "$transport_vhost" != "watcher-rpc" ]; then' - ' echo "Expected watcher-transport vhost ''watcher-rpc'', found ''$transport_vhost''"' - ' exit 1' - ' fi' - ' echo "Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)"' - ' ' - ' # Verify notification-transport has correct user and vhost' - ' notif_user=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath=''{.spec.username}'')' - ' notif_vhost=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')' - ' if [ "$notif_user" != "watcher-notifications" ]; then' - ' echo "Expected notification-transport username ''watcher-notifications'', found ''$notif_user''"' - ' exit 1' - ' fi' - ' if [ "$notif_vhost" != "watcher-notifications" ]; then' - ' echo "Expected notification-transport vhost ''watcher-notifications'', found ''$notif_vhost''"' - ' exit 1' - ' fi' - ' echo "Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)"' - ' ' - ' # Verify that watcher.conf contains the notifications transport_url' - ' WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l "service=watcher-api" -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)' - ' if [ -z "${WATCHER_API_POD}" ]; then' - ' echo "No watcher-api pod found"' - ' exit 1' - ' fi' - ' # Verify RPC transport_url in DEFAULT section' - ' rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E ''^\[DEFAULT\]'' -A 50 | grep ''transport_url'' | head -1 || true)' - ' if [ -z "$rpc_transport_url" ]; then' - ' echo "transport_url not found in DEFAULT section"' - ' exit 1' - ' fi' - ' echo "Found RPC transport_url: $rpc_transport_url"' - ' ' - ' # Verify the RPC transport_url contains the correct vhost (watcher-rpc)' - ' if ! echo "$rpc_transport_url" | grep -q ''/watcher-rpc''; then' - ' echo "RPC transport_url does not contain expected vhost ''/watcher-rpc''"' - ' exit 1' - ' fi' - ' echo "Successfully verified vhost ''watcher-rpc'' in RPC transport_url"' - ' ' - ' # Verify the RPC transport_url contains the correct username (watcher-rpc)' - ' if ! echo "$rpc_transport_url" | grep -q ''watcher-rpc:''; then' - ' echo "RPC transport_url does not contain expected username ''watcher-rpc:''"' - ' exit 1' - ' fi' - ' echo "Successfully verified username ''watcher-rpc'' in RPC transport_url"' - ' ' - ' # Verify oslo_messaging_notifications section has transport_url configured' - ' notif_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -A 5 ''\[oslo_messaging_notifications\]'' | grep ''transport_url'' || true)' - ' if [ -z "$notif_transport_url" ]; then' - ' echo "transport_url not found in oslo_messaging_notifications section"' - ' exit 1' - ' fi' - ' echo "Found notifications transport_url: $notif_transport_url"' - ' ' - ' # Verify the notifications transport_url contains the correct vhost (watcher-notifications)' - ' if ! echo "$notif_transport_url" | grep -q ''/watcher-notifications''; then' - ' echo "Notifications transport_url does not contain expected vhost ''/watcher-notifications''"' - ' exit 1' - ' fi' - ' echo "Successfully verified vhost ''watcher-notifications'' in notifications transport_url"' - ' ' - ' # Verify the notifications transport_url contains the correct username (watcher-notifications)' - ' if ! echo "$notif_transport_url" | grep -q ''watcher-notifications:''; then' - ' echo "Notifications transport_url does not contain expected username ''watcher-notifications:''"' - ' exit 1' - ' fi' - ' echo "Successfully verified username ''watcher-notifications'' in notifications transport_url"' - ' ' - ' exit 0' - ' ]' - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready watcher/watcher-kuttl -n watcher-kuttl-default --timeout=300s' - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | watcher.watcher.openstack.org/watcher-kuttl condition met' - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl -n watcher-kuttl-default -o ''jsonpath={.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}''' - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + grep -q True' - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | + echo ''WatcherNotificationTransportURLReady condition is True''' - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady condition is True' - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport' - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | ++ kubectl get transporturl -n watcher-kuttl-default -o name' - ' logger.go:42: 19:05:14 | watcher-rmquser/1-deploy | ++ wc -l' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + transport_count=1' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ wc -l' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl -n watcher-kuttl-default -o name' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + notification_transport_count=1' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + echo ''Correctly found 2 TransportURLs (separate clusters: transport and notification)''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs (separate clusters: transport and notification)' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.username}''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + transport_user=watcher-rpc' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.vhost}''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + transport_vhost=watcher-rpc' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!='' watcher-rpc '']''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!='' watcher-rpc '']''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + echo ''Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default -o ''jsonpath={.spec.username}''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + notif_user=watcher-notifications' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default -o ''jsonpath={.spec.vhost}''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' watcher-notifications ''!='' watcher-notifications '']''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' watcher-notifications ''!='' watcher-notifications '']''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + echo ''Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep -v ''^$''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ head -1' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl get pods -n watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name --no-headers' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | + ''['' -z watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep -E ''^\[DEFAULT\]'' -A 50' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ grep transport_url' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:05:15 | watcher-rmquser/1-deploy | ++ head -1' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + rpc_transport_url=''transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url = rabbit://**********=1'' '']''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''Found RPC transport_url: transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Found RPC transport_url: transport_url = rabbit://**********=1' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''Successfully verified vhost ''\''''watcher-rpc''\'''' in RPC transport_url''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Successfully verified vhost ''watcher-rpc'' in RPC transport_url' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''Successfully verified username ''\''''watcher-rpc''\'''' in RPC transport_url''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Successfully verified username ''watcher-rpc'' in RPC transport_url' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | ++ grep transport_url' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | ++ grep -A 5 ''\[oslo_messaging_notifications\]''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + notif_transport_url=''transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url = rabbit://**********=1'' '']''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''Found notifications transport_url: transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Found notifications transport_url: transport_url = rabbit://**********=1' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''Successfully verified vhost ''\''''watcher-notifications''\'''' in notifications transport_url''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Successfully verified vhost ''watcher-notifications'' in notifications transport_url' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''transport_url = rabbit://**********=1''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + echo ''Successfully verified username ''\''''watcher-notifications''\'''' in notifications transport_url''' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | Successfully verified username ''watcher-notifications'' in notifications transport_url' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | + exit 0' - ' logger.go:42: 19:05:16 | watcher-rmquser/1-deploy | test step completed 1-deploy' - ' logger.go:42: 19:05:16 | watcher-rmquser/2-cleanup-watcher | starting test step 2-cleanup-watcher' - ' logger.go:42: 19:05:22 | watcher-rmquser/2-cleanup-watcher | test step completed 2-cleanup-watcher' - ' logger.go:42: 19:05:22 | watcher-rmquser | skipping kubernetes event logging' - === CONT kuttl/harness/watcher - ' logger.go:42: 19:05:22 | watcher | Skipping creation of user-supplied namespace: watcher-kuttl-default' - ' logger.go:42: 19:05:22 | watcher/0-cleanup-watcher | starting test step 0-cleanup-watcher' - ' logger.go:42: 19:05:22 | watcher/0-cleanup-watcher | test step completed 0-cleanup-watcher' - ' logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | starting test step 1-deploy-with-defaults' - ' logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | Watcher:watcher-kuttl-default/watcher-kuttl created' - ' logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" -ge 1 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:05:22 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:05:24 | watcher/1-deploy-with-defaults | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:05:25 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" -ge 1 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:05:25 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:05:25 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:05:28 | watcher/1-deploy-with-defaults | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:05:29 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" -ge 1 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:05:29 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:05:29 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | ++ grep watcher' - ' logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' - ' logger.go:42: 19:05:31 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136' - ' logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | + ''['' 5be153439946473bb5b269e3a0a3e136 == 5be153439946473bb5b269e3a0a3e136 '']''' - ' logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:05:33 | watcher/1-deploy-with-defaults | + ''['' -n '''' '']''' - ' logger.go:42: 19:05:34 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" -ge 1 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:05:34 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:05:34 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' - ' logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | ++ grep watcher' - ' logger.go:42: 19:05:37 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + ''['' 5be153439946473bb5b269e3a0a3e136 == 5be153439946473bb5b269e3a0a3e136 '']''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ base64 -d' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ base64 -d' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ base64 -d' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name = regionOne''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:05:39 | watcher/1-deploy-with-defaults | + exit 0' - ' logger.go:42: 19:05:41 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" -ge 1 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:05:41 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:05:41 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' - ' logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | ++ grep watcher' - ' logger.go:42: 19:05:43 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + ''['' 5be153439946473bb5b269e3a0a3e136 == 5be153439946473bb5b269e3a0a3e136 '']''' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ base64 -d' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ base64 -d' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ base64 -d' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name = regionOne''' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']''' - ' logger.go:42: 19:05:45 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:05:46 | watcher/1-deploy-with-defaults | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:05:46 | watcher/1-deploy-with-defaults | + exit 0' - ' logger.go:42: 19:05:47 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" -ge 1 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:05:47 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:05:47 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' - ' logger.go:42: 19:05:49 | watcher/1-deploy-with-defaults | ++ grep watcher' - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136' - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + ''['' 5be153439946473bb5b269e3a0a3e136 == 5be153439946473bb5b269e3a0a3e136 '']''' - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1' - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ base64 -d' - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ base64 -d' - ' logger.go:42: 19:05:51 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | ++ base64 -d' - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name = regionOne''' - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']''' - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:05:52 | watcher/1-deploy-with-defaults | + exit 0' - ' logger.go:42: 19:05:53 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 ]' - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" -ge 1 ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:05:53 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:05:53 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' - ' logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | ++ grep watcher' - ' logger.go:42: 19:05:55 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | + SERVICEID=5be153439946473bb5b269e3a0a3e136' - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | + ''['' 5be153439946473bb5b269e3a0a3e136 == 5be153439946473bb5b269e3a0a3e136 '']''' - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ base64 -d' - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1' - ' logger.go:42: 19:05:57 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ base64 -d' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name = regionOne''' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ base64 -d' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']''' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | + exit 0' - ' logger.go:42: 19:05:58 | watcher/1-deploy-with-defaults | test step completed 1-deploy-with-defaults' - ' logger.go:42: 19:05:58 | watcher/2-cleanup-watcher | starting test step 2-cleanup-watcher' - ' logger.go:42: 19:05:58 | watcher/2-cleanup-watcher | test step completed 2-cleanup-watcher' - ' logger.go:42: 19:05:58 | watcher/3-precreate-mariadbaccount | starting test step 3-precreate-mariadbaccount' - ' logger.go:42: 19:05:58 | watcher/3-precreate-mariadbaccount | MariaDBAccount:watcher-kuttl-default/watcher-precreated created' - ' logger.go:42: 19:05:58 | watcher/3-precreate-mariadbaccount | test step completed 3-precreate-mariadbaccount' - ' logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | starting test step 4-deploy-with-precreated-account' - ' logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | Secret:wa**********ig created' - ' logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | Watcher:watcher-kuttl-default/watcher-kuttl created' - ' logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:05:58 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | error: Internal error occurred: error executing command in container: container is not created or running' - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | ++ echo' - ' logger.go:42: 19:05:59 | watcher/4-deploy-with-precreated-account | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:00 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:01 | watcher/4-deploy-with-precreated-account | + APIPOD=' - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:02 | watcher/4-deploy-with-precreated-account | + APIPOD=' - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:03 | watcher/4-deploy-with-precreated-account | + APIPOD=' - ' logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:04 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:05 | watcher/4-deploy-with-precreated-account | + APIPOD=' - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:06 | watcher/4-deploy-with-precreated-account | + APIPOD=' - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:07 | watcher/4-deploy-with-precreated-account | + APIPOD=' - ' logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:08 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:09 | watcher/4-deploy-with-precreated-account | + APIPOD=' - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:10 | watcher/4-deploy-with-precreated-account | + APIPOD=' - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:11 | watcher/4-deploy-with-precreated-account | + APIPOD=' - ' logger.go:42: 19:06:12 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:12 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:13 | watcher/4-deploy-with-precreated-account | + APIPOD=' - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:14 | watcher/4-deploy-with-precreated-account | + APIPOD=' - ' logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:15 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | error: unable to upgrade connection: container not found ("watcher-api")' - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | ++ echo' - ' logger.go:42: 19:06:16 | watcher/4-deploy-with-precreated-account | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:17 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:18 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:19 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:20 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:21 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:21 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:22 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:23 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:23 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:24 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:25 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:26 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:27 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:28 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:29 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:31 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:32 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:32 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:33 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:34 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:35 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:36 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:37 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:38 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:40 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:41 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:41 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:42 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:43 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:44 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:45 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:47 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:48 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:49 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:50 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:51 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:52 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:53 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:54 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:56 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:57 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:06:58 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:06:59 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:07:00 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:07:01 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:07:03 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:07:04 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:07:04 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:07:05 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:07:06 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | running command: [sh -c set -euxo pipefail' - ' oc project watcher-kuttl-default' - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" -ocustom-columns=:metadata.name|grep -v ^$|head -1)' - ' if [ -n "${APIPOD}" ]; then' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) |grep -c "^# Global config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c "^# Service config") == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') == 1 ]' - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) |grep -czPo ''TimeOut 80'') == 1 ]' - ' else' - ' exit 1' - ' fi' - ' ]' - ' logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | + oc project watcher-kuttl-default' - ' logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' - ' logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' - ' logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | ++ head -1' - ' logger.go:42: 19:07:07 | watcher/4-deploy-with-precreated-account | ++ grep -v ''^$''' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + ''['' -n watcher-kuttl-api-0 '']''' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Global config''' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Global config' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ grep -c ''^# Service config''' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' Service config' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://**********=1'' control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log ''#'' empty notification_level means that no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_test:500520b2ea4cb7ae05a658978fc23911@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt memcache_tls_enabled = true project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[watcher_clients_auth]'' project_domain_name = Default project_name = service user_domain_name = Default password = pa**********rd username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type = internal region_name = regionOne ''[placement_client]'' interface = internal region_name = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ grep -czPo ''TimeOut 80''' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | ++ echo ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess internal display-name=internal group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin '''' Options -Indexes +FollowSymLinks +MultiViews AllowOverride None Require all granted '''' ''##'' Logging ErrorLog /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:07:08 | watcher/4-deploy-with-precreated-account | test step completed 4-deploy-with-precreated-account' - ' logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | starting test step 5-cleanup-watcher' - ' logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | running command: [sh -c set -ex' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' - ' ]' - ' logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:07:08 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher''' - ' logger.go:42: 19:07:11 | watcher/5-cleanup-watcher | + ''['' 1 == 0 '']''' - ' logger.go:42: 19:07:14 | watcher/5-cleanup-watcher | running command: [sh -c set -ex' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' - ' ]' - ' logger.go:42: 19:07:14 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:07:14 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher''' - ' logger.go:42: 19:07:16 | watcher/5-cleanup-watcher | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:07:16 | watcher/5-cleanup-watcher | test step completed 5-cleanup-watcher' - ' logger.go:42: 19:07:16 | watcher | skipping kubernetes event logging' - === CONT kuttl/harness/watcher-cinder - ' logger.go:42: 19:07:16 | watcher-cinder | Skipping creation of user-supplied namespace: watcher-kuttl-default' - ' logger.go:42: 19:07:16 | watcher-cinder/0-cleanup-watcher | starting test step 0-cleanup-watcher' - ' logger.go:42: 19:07:16 | watcher-cinder/0-cleanup-watcher | test step completed 0-cleanup-watcher' - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | starting test step 1-deploy-watcher-no-cinder' - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | Watcher:watcher-kuttl-default/watcher-kuttl created' - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in namespace "watcher-kuttl-default"' - ' logger.go:42: 19:07:16 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in namespace "watcher-kuttl-default"' - ' logger.go:42: 19:07:17 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in namespace "watcher-kuttl-default"' - ' logger.go:42: 19:07:18 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:19 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:19 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:19 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:20 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in namespace "watcher-kuttl-default"' - ' logger.go:42: 19:07:20 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in namespace "watcher-kuttl-default"' - ' logger.go:42: 19:07:21 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in namespace "watcher-kuttl-default"' - ' logger.go:42: 19:07:22 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in namespace "watcher-kuttl-default"' - ' logger.go:42: 19:07:23 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in namespace "watcher-kuttl-default"' - ' logger.go:42: 19:07:24 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in namespace "watcher-kuttl-default"' - ' logger.go:42: 19:07:25 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in namespace "watcher-kuttl-default"' - ' logger.go:42: 19:07:26 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | Error from server (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" is waiting to start: ContainerCreating' - ' logger.go:42: 19:07:28 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | Error from server (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" is waiting to start: ContainerCreating' - ' logger.go:42: 19:07:29 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:30 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:31 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:32 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:07:33 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:34 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:35 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:36 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:37 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:07:38 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:39 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:07:40 | watcher-cinder/1-deploy-watcher-no-cinder | test step completed 1-deploy-watcher-no-cinder' - ' logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | starting test step 2-deploy-cinder' - ' logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | OpenStackControlPlane:watcher-kuttl-default/openstack updated' - ' logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:40 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:41 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:42 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:43 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:43 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:43 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:44 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:45 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:46 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:47 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:48 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:48 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:48 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:49 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:50 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:51 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:52 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:53 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:54 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:54 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:54 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:55 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:56 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:57 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:58 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:07:59 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:00 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:00 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:00 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:01 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:02 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:03 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:04 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:05 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:06 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:06 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:06 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:07 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:08 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:09 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:10 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:11 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:12 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:12 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:12 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:13 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:14 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:15 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:16 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:17 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:17 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:17 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:18 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:19 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:20 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:21 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:22 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:23 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:23 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:23 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:24 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:25 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:26 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:27 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' - ' logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | Error from server (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" is waiting to start: ContainerCreating' - ' logger.go:42: 19:08:28 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:08:29 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:29 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:29 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:30 | watcher-cinder/2-deploy-cinder | Error from server (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" is waiting to start: ContainerCreating' - ' logger.go:42: 19:08:30 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:31 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:32 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:33 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:34 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:08:35 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:35 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:35 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:36 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:37 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:38 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:39 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:08:40 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision detects that there is a cinder service and' - ' # does not log that storage collector is skipped' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 0 ]' - ' ]' - ' logger.go:42: 19:08:40 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:40 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:41 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:08:41 | watcher-cinder/2-deploy-cinder | test step completed 2-deploy-cinder' - ' logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | starting test step 3-remove-cinder' - ' logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | OpenStackControlPlane:watcher-kuttl-default/openstack updated' - ' logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:41 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:42 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:43 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:08:44 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:44 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:44 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:45 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:46 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | Error from server (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" is waiting to start: ContainerCreating' - ' logger.go:42: 19:08:48 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | Error from server (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" is waiting to start: ContainerCreating' - ' logger.go:42: 19:08:49 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:50 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:51 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' - ' logger.go:42: 19:08:52 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:52 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:52 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:53 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:54 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:55 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:56 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:57 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:08:58 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:08:58 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:08:58 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:08:59 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail' - ' # check that the decision engine correctly detects that there is no cinder service' - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block storage service is not enabled, skipping storage collector'')" == 2 ]' - ' ]' - ' logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' - ' logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block storage service is not enabled, skipping storage collector''' - ' logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' - ' logger.go:42: 19:09:00 | watcher-cinder/3-remove-cinder | test step completed 3-remove-cinder' - ' logger.go:42: 19:09:00 | watcher-cinder/4-cleanup-watcher | starting test step 4-cleanup-watcher' - ' logger.go:42: 19:09:08 | watcher-cinder/4-cleanup-watcher | test step completed 4-cleanup-watcher' - ' logger.go:42: 19:09:08 | watcher-cinder | skipping kubernetes event logging' - === CONT kuttl/harness/watcher-api-scaling - ' logger.go:42: 19:09:08 | watcher-api-scaling | Skipping creation of user-supplied namespace: watcher-kuttl-default' - ' logger.go:42: 19:09:08 | watcher-api-scaling/0-cleanup-watcher | starting test step 0-cleanup-watcher' - ' logger.go:42: 19:09:08 | watcher-api-scaling/0-cleanup-watcher | test step completed 0-cleanup-watcher' - ' logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | starting test step 1-deploy-with-defaults' - ' logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | Watcher:watcher-kuttl-default/watcher-kuttl created' - ' logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:09:08 | watcher-api-scaling/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:09:11 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:09:12 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:09:12 | watcher-api-scaling/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:09:12 | watcher-api-scaling/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:09:14 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 0 == 1 '']''' - ' logger.go:42: 19:09:15 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:09:15 | watcher-api-scaling/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:09:15 | watcher-api-scaling/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults | ++ awk ''{print $1}''' - ' logger.go:42: 19:09:18 | watcher-api-scaling/1-deploy-with-defaults | ++ grep watcher' - ' logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=63195c8d1c084032a109516d802e4c7d' - ' logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d '']''' - ' logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:09:20 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:09:21 | watcher-api-scaling/1-deploy-with-defaults | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:09:21 | watcher-api-scaling/1-deploy-with-defaults | + exit 0' - ' logger.go:42: 19:09:22 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:09:22 | watcher-api-scaling/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:09:22 | watcher-api-scaling/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults | ++ grep watcher' - ' logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults | ++ awk ''{print $1}''' - ' logger.go:42: 19:09:24 | watcher-api-scaling/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=63195c8d1c084032a109516d802e4c7d' - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d '']''' - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:09:26 | watcher-api-scaling/1-deploy-with-defaults | + exit 0' - ' logger.go:42: 19:09:27 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:09:27 | watcher-api-scaling/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:09:27 | watcher-api-scaling/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults | ++ awk ''{print $1}''' - ' logger.go:42: 19:09:29 | watcher-api-scaling/1-deploy-with-defaults | ++ grep watcher' - ' logger.go:42: 19:09:31 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=63195c8d1c084032a109516d802e4c7d' - ' logger.go:42: 19:09:31 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d '']''' - ' logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:09:32 | watcher-api-scaling/1-deploy-with-defaults | + exit 0' - ' logger.go:42: 19:09:33 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]' - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" ]' - ' # If we are running the container locally, skip following test' - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" == "" ]; then' - ' exit 0' - ' fi' - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)' - ' counter=0' - ' for i in ${env_variables}; do' - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' - ' echo ${i}' - ' counter=$((counter + 1))' - ' fi' - ' done' - ' if [ ${counter} -lt 3 ]; then' - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' - ' exit 1' - ' else' - ' echo "Success: ${counter} _URL_DEFAULT variables found."' - ' fi' - ' ]' - ' logger.go:42: 19:09:33 | watcher-api-scaling/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:09:33 | watcher-api-scaling/1-deploy-with-defaults | ++ grep -c ''^watcher''' - ' logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 1 == 1 '']''' - ' logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults | ++ grep watcher' - ' logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults | ++ awk ''{print $1}''' - ' logger.go:42: 19:09:35 | watcher-api-scaling/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type -c ID' - ' logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=63195c8d1c084032a109516d802e4c7d' - ' logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' - ' logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 63195c8d1c084032a109516d802e4c7d == 63195c8d1c084032a109516d802e4c7d '']''' - ' logger.go:42: 19:09:37 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' - ' logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | + ''['' -n n5dfhc5h7ch87h5f7h68h64bhc8h5d6h686h58chbfh667h659h65fh686h5bhf4h565h668h95h56ch5bch599h65fh67ch668h54fh688h55fhc4h679q '']''' - ' logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' - ' logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | + ''['' '''' == '''' '']''' - ' logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | + exit 0' - ' logger.go:42: 19:09:38 | watcher-api-scaling/1-deploy-with-defaults | test step completed 1-deploy-with-defaults' - ' logger.go:42: 19:09:38 | watcher-api-scaling/2-scale-up-watcher-api | starting test step 2-scale-up-watcher-api' - ' logger.go:42: 19:09:38 | watcher-api-scaling/2-scale-up-watcher-api | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": "replace", "path": "/spec/apiServiceTemplate/replicas", "value":3}]''' - ' ]' - ' logger.go:42: 19:09:38 | watcher-api-scaling/2-scale-up-watcher-api | watcher.watcher.openstack.org/watcher-kuttl patched' - ' logger.go:42: 19:09:49 | watcher-api-scaling/2-scale-up-watcher-api | test step completed 2-scale-up-watcher-api' - ' logger.go:42: 19:09:49 | watcher-api-scaling/3-scale-down-watcher-api | starting test step 3-scale-down-watcher-api' - ' logger.go:42: 19:09:49 | watcher-api-scaling/3-scale-down-watcher-api | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": "replace", "path": "/spec/apiServiceTemplate/replicas", "value":1}]''' - ' ]' - ' logger.go:42: 19:09:49 | watcher-api-scaling/3-scale-down-watcher-api | watcher.watcher.openstack.org/watcher-kuttl patched' - ' logger.go:42: 19:09:52 | watcher-api-scaling/3-scale-down-watcher-api | test step completed 3-scale-down-watcher-api' - ' logger.go:42: 19:09:52 | watcher-api-scaling/4-scale-down-zero-watcher-api | starting test step 4-scale-down-zero-watcher-api' - ' logger.go:42: 19:09:52 | watcher-api-scaling/4-scale-down-zero-watcher-api | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": "replace", "path": "/spec/apiServiceTemplate/replicas", "value":0}]''' - ' ]' - ' logger.go:42: 19:09:52 | watcher-api-scaling/4-scale-down-zero-watcher-api | watcher.watcher.openstack.org/watcher-kuttl patched' - ' logger.go:42: 19:09:53 | watcher-api-scaling/4-scale-down-zero-watcher-api | test step completed 4-scale-down-zero-watcher-api' - ' logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | starting test step 5-cleanup-watcher' - ' logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | running command: [sh -c set -ex' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' - ' ]' - ' logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:09:53 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c ''^watcher''' - ' logger.go:42: 19:09:56 | watcher-api-scaling/5-cleanup-watcher | + ''['' 1 == 0 '']''' - ' logger.go:42: 19:09:57 | watcher-api-scaling/5-cleanup-watcher | running command: [sh -c set -ex' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' - ' ]' - ' logger.go:42: 19:09:57 | watcher-api-scaling/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:09:57 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c ''^watcher''' - ' logger.go:42: 19:09:59 | watcher-api-scaling/5-cleanup-watcher | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:10:00 | watcher-api-scaling/5-cleanup-watcher | running command: [sh -c set -ex' - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' - ' ]' - ' logger.go:42: 19:10:00 | watcher-api-scaling/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name -c Type' - ' logger.go:42: 19:10:00 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c ''^watcher''' - ' logger.go:42: 19:10:02 | watcher-api-scaling/5-cleanup-watcher | + ''['' 0 == 0 '']''' - ' logger.go:42: 19:10:02 | watcher-api-scaling/5-cleanup-watcher | test step completed 5-cleanup-watcher' - ' logger.go:42: 19:10:02 | watcher-api-scaling | skipping kubernetes event logging' - === CONT kuttl/harness/deps - ' logger.go:42: 19:10:02 | deps | Ignoring infra.yaml as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' - ' logger.go:42: 19:10:02 | deps | Ignoring keystone.yaml as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' - ' logger.go:42: 19:10:02 | deps | Ignoring kustomization.yaml as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' - ' logger.go:42: 19:10:02 | deps | Ignoring namespace.yaml as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' - ' logger.go:42: 19:10:02 | deps | Ignoring telemetry.yaml as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' - ' logger.go:42: 19:10:02 | deps | Skipping creation of user-supplied namespace: watcher-kuttl-default' - ' logger.go:42: 19:10:02 | deps | skipping kubernetes event logging' - === CONT kuttl/harness/common - ' logger.go:42: 19:10:02 | common | Ignoring cleanup-assert.yaml as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' - ' logger.go:42: 19:10:02 | common | Ignoring cleanup-errors.yaml as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' - ' logger.go:42: 19:10:02 | common | Ignoring cleanup-watcher.yaml as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' - ' logger.go:42: 19:10:02 | common | Ignoring deploy-with-defaults.yaml as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' - ' logger.go:42: 19:10:02 | common | Skipping creation of user-supplied namespace: watcher-kuttl-default' - ' logger.go:42: 19:10:02 | common | skipping kubernetes event logging' - === NAME kuttl - ' harness.go:406: run tests finished' - ' harness.go:514: cleaning up' - ' harness.go:571: removing temp folder: ""' - '--- FAIL: kuttl (1502.65s)' - ' --- FAIL: kuttl/harness (0.00s)' - ' --- FAIL: kuttl/harness/appcred-tests (959.40s)' - ' --- PASS: kuttl/harness/watcher-notification (36.28s)' - ' --- PASS: kuttl/harness/watcher-topology (33.59s)' - ' --- PASS: kuttl/harness/watcher-tls-certs-change (37.89s)' - ' --- PASS: kuttl/harness/watcher-tls (120.05s)' - ' --- PASS: kuttl/harness/watcher-rmquser (34.79s)' - ' --- PASS: kuttl/harness/watcher (114.14s)' - ' --- PASS: kuttl/harness/watcher-cinder (112.06s)' - ' --- PASS: kuttl/harness/watcher-api-scaling (54.41s)' - ' --- PASS: kuttl/harness/deps (0.00s)' - ' --- PASS: kuttl/harness/common (0.00s)' - FAIL 2026-03-09 19:10:03,959 p=40838 u=zuul n=ansible | NO MORE HOSTS LEFT ************************************************************* 2026-03-09 19:10:03,960 p=40838 u=zuul n=ansible | PLAY RECAP ********************************************************************* 2026-03-09 19:10:03,960 p=40838 u=zuul n=ansible | controller : ok=2 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 2026-03-09 19:10:03,960 p=40838 u=zuul n=ansible | Monday 09 March 2026 19:10:03 +0000 (0:25:04.305) 0:25:04.762 ********** 2026-03-09 19:10:03,960 p=40838 u=zuul n=ansible | =============================================================================== 2026-03-09 19:10:03,960 p=40838 u=zuul n=ansible | run kuttl test suite from operator Makefile -------------------------- 1504.31s 2026-03-09 19:10:03,960 p=40838 u=zuul n=ansible | run_hook : Loop on hooks for pre_kuttl_from_operator -------------------- 0.16s 2026-03-09 19:10:03,960 p=40838 u=zuul n=ansible | run_hook : Assert single hooks are all mappings ------------------------- 0.15s 2026-03-09 19:10:03,960 p=40838 u=zuul n=ansible | run_hook : Assert parameters are valid ---------------------------------- 0.07s 2026-03-09 19:10:03,960 p=40838 u=zuul n=ansible | Run hooks before running kuttl tests ------------------------------------ 0.03s 2026-03-09 19:10:03,960 p=40838 u=zuul n=ansible | Monday 09 March 2026 19:10:03 +0000 (0:25:04.305) 0:25:04.762 ********** 2026-03-09 19:10:03,960 p=40838 u=zuul n=ansible | =============================================================================== 2026-03-09 19:10:03,960 p=40838 u=zuul n=ansible | ansible.builtin.command ---------------------------------------------- 1504.31s 2026-03-09 19:10:03,961 p=40838 u=zuul n=ansible | run_hook ---------------------------------------------------------------- 0.38s 2026-03-09 19:10:03,961 p=40838 u=zuul n=ansible | ansible.builtin.include_role -------------------------------------------- 0.03s 2026-03-09 19:10:03,961 p=40838 u=zuul n=ansible | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2026-03-09 19:10:03,961 p=40838 u=zuul n=ansible | total ---------------------------------------------------------------- 1504.72s